xfsprogs: everything headed towards 6.9
PATCH v2 1/1 xfsprogs: everything headed towards 6.9
Hi Carlos,
Now that 6.9-rc1 is out, here's v2 of my earlier patchbomb to help us get xfsprogs caught up to 6.8. There are four new bugfixes for a 6.8 release, and I've added my libxfs-6.9-sync branch + all the 6.9 changes that I've queued up so far.
Sorry about the giant patchset tho. :(
--D
Quoted 8 lines · Darrick J. Wong
Hi Carlos,
Now that 6.9-rc1 is out, here's v2 of my earlier patchbomb to help us get xfsprogs caught up to 6.8. There are four new bugfixes for a 6.8 release, and I've added my libxfs-6.9-sync branch + all the 6.9 changes that I've queued up so far.
Sorry about the giant patchset tho. :(
Unreviewed patches:
[PATCHSET 04/18] xfsprogs: bug fixes for 6.8:
patches 2-5
[PATCHSET 11/18] libxfs: sync with 6.9:
patches 88-90, 92
[PATCHSET v29.4 12/18] xfsprogs: bmap log intent cleanups [PATCHSET v29.4 13/18] xfsprogs: widen BUI formats to support [PATCHSET v29.4 14/18] xfs_spaceman: updates for 6.9 [PATCHSET v29.4 15/18] xfs_scrub: updates for 6.9 [PATCHSET v29.4 16/18] xfs_repair: use in-memory rmap btrees [PATCHSET v29.4 17/18] xfs_repair: reduce refcount repair memory [PATCHSET v29.4 18/18] mkfs: cleanups for 6.9
all patches in these series
--D
PATCH 1/18 xfsprogs: convert utilities to use new rt helpers
In reply to Darrick J. WongHi all,
The patches in this series clean up a lot of realtime space usage code the userspace utilities. This involves correcting incorrect type usage, renaming variables to reflect their actual usage; and converting open code logic to use the new helpers that were just added to libxfs.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=fix-realtime-units
---
Commits in this patchset:
* libxfs: fix incorrect porting to 6.7
* mkfs: fix log sunit rounding when external logs are in use
* xfs_repair: fix confusing rt space units in the duplicate detection code
* libxfs: create a helper to compute leftovers of realtime extents
* libxfs: use helpers to convert rt block numbers to rt extent numbers
* xfs_repair: convert utility to use new rt extent helpers and types
* mkfs: convert utility to use new rt extent helpers and types
* xfs_{db,repair}: convert open-coded xfs_rtword_t pointer accesses to helper
* xfs_repair: convert helpers for rtbitmap block/wordcount computations
* xfs_{db,repair}: use accessor functions for bitmap words
* xfs_{db,repair}: use helpers for rtsummary block/wordcount computations
* xfs_{db,repair}: use accessor functions for summary info words
* xfs_{db,repair}: use m_blockwsize instead of sb_blocksize for rt blocks
--- db/check.c | 90 ++++++++++++++++++++++++++++++++++++---------- include/libxfs.h | 4 ++ libxfs/Makefile | 1 + libxfs/init.c | 8 ++-- libxfs/libxfs_api_defs.h | 8 ++++ libxfs/logitem.c | 3 +- libxfs/trans.c | 3 +- libxfs/xfs_rtbitmap.c | 2 + libxfs/xfs_rtbitmap.h | 3 -- mkfs/proto.c | 41 ++++++++++++++------- mkfs/xfs_mkfs.c | 16 ++++++-- repair/agheader.h | 2 + repair/dinode.c | 21 ++++++----- repair/globals.c | 4 +- repair/globals.h | 4 +- repair/incore.c | 16 ++++---- repair/incore.h | 15 +++----- repair/incore_ext.c | 74 ++++++++++++++++++++------------------ repair/phase4.c | 16 ++++---- repair/phase6.c | 28 +++++++++++--- repair/rt.c | 64 ++++++++++++++++++++++----------- repair/rt.h | 6 +-- repair/scan.c | 2 + 23 files changed, 278 insertions(+), 153 deletions(-)
PATCH 1/13 libxfs: fix incorrect porting to 6.7
From: Darrick J. Wong <[email protected]>
Userspace libxfs is supposed to match the kernel libxfs except for the preprocessor include directives. Fix a few discrepancies that came up for whatever reason.
To fix the build errors resulting from CONFIG_XFS_RT not being defined, add it to libxfs.h and alter the Makefile to track xfs_rtbitmap.h.
--- db/check.c | 1 - include/libxfs.h | 4 ++++ libxfs/Makefile | 1 + libxfs/xfs_rtbitmap.c | 2 +- libxfs/xfs_rtbitmap.h | 3 --- repair/rt.c | 1 - 6 files changed, 6 insertions(+), 6 deletions(-)
db/check.c +0 -1
@@ -20,7 +20,6 @@#include "init.h" #include "malloc.h" #include "dir2.h" -#include "xfs_rtbitmap.h" typedef enum { IS_USER_QUOTA, IS_PROJECT_QUOTA, IS_GROUP_QUOTA,
include/libxfs.h +4 -0
@@ -7,6 +7,9 @@#ifndef __LIBXFS_H__#define __LIBXFS_H__ +/* For userspace XFS_RT is always defined */ +#define CONFIG_XFS_RT +#include "libxfs_api_defs.h" #include "platform_defs.h" #include "xfs.h"
@@ -80,6 +83,7 @@ struct iomap;#include "xfs_refcount_btree.h" #include "xfs_refcount.h" #include "xfs_btree_staging.h" +#include "xfs_rtbitmap.h" #ifndef ARRAY_SIZE #define ARRAY_SIZE (x) (sizeof(x) / sizeof((x)[0]))
libxfs/Makefile +1 -0
@@ -50,6 +50,7 @@ HFILES = \ xfs_refcount_btree.h \ xfs_rmap.h \ xfs_rmap_btree.h \ + xfs_rtbitmap.h \ xfs_sb.h \ xfs_shared.h \ xfs_trans_resv.h \
libxfs/xfs_rtbitmap.c +1 -1
@@ -931,7 +931,7 @@ xfs_rtcheck_alloc_range(*/ int xfs_rtfree_extent ( -xfs_trans_t *tp,/* transaction pointer */ +struct xfs_trans *tp,/* transaction pointer */ xfs_rtxnum_t start,/* starting rtext number to free */ xfs_rtxlen_t len)/* length of extent freed */ {
libxfs/xfs_rtbitmap.h +0 -3
@@ -6,9 +6,6 @@ #ifndef __XFS_RTBITMAP_H__ #define __XFS_RTBITMAP_H__ -/* For userspace XFS_RT is always defined */ -#define CONFIG_XFS_RT - struct xfs_rtalloc_args { struct xfs_mount *mp; struct xfs_trans *tp;
repair/rt.c +0 -1
@@ -13,7 +13,6 @@#include "protos.h" #include "err_protos.h" #include "rt.h" -#include "xfs_rtbitmap.h" #define xfs_highbit64 libxfs_highbit64/* for XFS_RTBLOCKLOG macro */
PATCH 2/13 mkfs: fix log sunit rounding when external logs are in use
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Due to my heinous nature, I set up an external log device with 4k LBAs using this command:
# losetup -b 4096 -o 4096 --sizelimit $(( (128 * 1048576) - 4096 )) -f /dev/sdb # blockdev --getsize64 /dev/loop0 134213632
This creates a log device that is slightly smaller than 128MB in size. Next I ran generic/054, which sets the log sunit to 256k and fails:
# mkfs.xfs -f /dev/sda -l logdev=/dev/loop0,su=256k,version=2 -s size=4096
meta-data=/dev/sda isize=512 agcount=4, agsize=72448 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=1
= metadir=0
data = bsize=4096 blocks=289792, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0
log =/dev/loop0 bsize=4096 blocks=32768, version=2
= sectsz=4096 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
= rgcount=0 rgsize=0 blks
Discarding blocks...Done.
Discarding blocks...Done.
mkfs.xfs: libxfs_device_zero write failed: No space left on deviceNotice that mkfs thinks it should format a 32768-fsblock external log, but the log device itself is 32767 fsblocks. Hence the write goes off the end of the device and we get ENOSPC.
I tracked this behavior down to align_log_size in mkfs, which first tries to round the log size up to a stripe boundary, then tries to round it down. Unfortunately, in the case of an external log we call the function with XFS_MAX_LOG_BLOCKS without accounting for the possibility that the log device might be smaller.
Correct the callsite and clean up the open-coded rounding.
--- mkfs/xfs_mkfs.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
mkfs/xfs_mkfs.c +11 -5
@@ -3338,13 +3338,13 @@ _("log size %lld is not a multiple of the log stripe unit %d\n"),usage (); } - tmp_logblocks = ((cfg->logblocks + (sunit -1 )) / sunit) * sunit; + tmp_logblocks =roundup_64 (cfg->logblocks, sunit);/* If the log is too large, round down instead of round up */ if ((tmp_logblocks >XFS_MAX_LOG_BLOCKS ) || ((tmp_logblocks << cfg->blocklog) >XFS_MAX_LOG_BYTES ) || tmp_logblocks > max_logblocks) { - tmp_logblocks = (cfg->logblocks / sunit) * sunit; + tmp_logblocks =rounddown_64 (cfg->logblocks, sunit); } cfg->logblocks = tmp_logblocks; }
@@ -3465,6 +3465,7 @@ static voidcalculate_log_size ( struct mkfs_params *cfg, struct cli_params *cli, + struct libxfs_init *xi, struct xfs_mount *mp) { struct xfs_sb *sbp = &mp->m_sb;
@@ -3503,8 +3504,13 @@ _("external log device size %lld blocks too small, must be at least %lld blocks\ } cfg->logstart =0 ; cfg->logagno =0 ; -if (cfg->lsunit) -align_log_size (cfg, cfg->lsunit,XFS_MAX_LOG_BLOCKS ); +if (cfg->lsunit) { +uint64_t max_logblocks; + + max_logblocks =min (DTOBT (xi->log.size, cfg->blocklog), +XFS_MAX_LOG_BLOCKS ); +align_log_size (cfg, cfg->lsunit, max_logblocks); + }validate_log_size (cfg->logblocks, cfg->blocklog, min_logblocks);return ;
@@ -4257,7 +4263,7 @@ main(* With the mount set up, we can finally calculate the log size * constraints and do default size calculations and final validation */ -calculate_log_size (&cfg, &cli, mp); +calculate_log_size (&cfg, &cli, &xi, mp);finish_superblock_setup (&cfg, mp, sbp);
PATCH 3/13 xfs_repair: fix confusing rt space units in the duplicate detection code
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Christoph Hellwig stumbled over the crosslinked file data detection code in xfs_repair. While trying to make sense of his fixpatch, I realized that the variable names and unit types are very misleading.
The rt dup tree builder inserts records in units of realtime extents. One query of the rt dup tree passes in a realtime extent number, but one of them does not. Confusingly, all the variable names have "block" even though they really mean "extent". This makes a real difference for rextsize > 1 filesystems, though given the lack of complaints I'm guessing there aren't many users.
Clean up this whole mess by fixing the variable names of the duplicates tree and the state array to reflect the units that are stored in the data structure, and fix the buggy query code. Later on in this patchset we'll fix the variable types too.
This seems to have been broken since before the start of the git repo.
--- repair/incore.c | 16 ++++++----- repair/incore.h | 15 ++++------ repair/incore_ext.c | 74 +++++++++++++++++++++++++++------------------------ repair/phase4.c | 12 ++++---- 4 files changed, 59 insertions(+), 58 deletions(-)
repair/incore.c +8 -8
@@ -178,21 +178,21 @@ static size_t rt_bmap_size;*/ int get_rtbmap ( -xfs_rtblock_t bno) +xfs_rtblock_t rtx) { -return (*(rt_bmap + bno /XR_BB_NUM ) >> - ((bno %XR_BB_NUM ) *XR_BB )) &XR_BB_MASK ; +return (*(rt_bmap + rtx /XR_BB_NUM ) >> + ((rtx %XR_BB_NUM ) *XR_BB )) &XR_BB_MASK ; }void set_rtbmap ( -xfs_rtblock_t bno, +xfs_rtblock_t rtx,int state) { - *(rt_bmap + bno /XR_BB_NUM ) = - ((*(rt_bmap + bno /XR_BB_NUM ) & - (~((uint64_t )XR_BB_MASK << ((bno %XR_BB_NUM ) *XR_BB )))) | - (((uint64_t ) state) << ((bno %XR_BB_NUM ) *XR_BB ))); + *(rt_bmap + rtx /XR_BB_NUM ) = + ((*(rt_bmap + rtx /XR_BB_NUM ) & + (~((uint64_t )XR_BB_MASK << ((rtx %XR_BB_NUM ) *XR_BB )))) | + (((uint64_t ) state) << ((rtx %XR_BB_NUM ) *XR_BB ))); } static void
repair/incore.h +6 -9
@@ -28,8 +28,8 @@ void set_bmap_ext(xfs_agnumber_t agno, xfs_agblock_t agbno,int get_bmap_ext (xfs_agnumber_t agno,xfs_agblock_t agbno,xfs_agblock_t maxbno,xfs_extlen_t *blen); -void set_rtbmap (xfs_rtblock_t bno,int state); -int get_rtbmap (xfs_rtblock_t bno); +void set_rtbmap (xfs_rtblock_t rtx,int state); +int get_rtbmap (xfs_rtblock_t rtx);static inline void set_bmap (xfs_agnumber_t agno,xfs_agblock_t agbno,int state)
@@ -70,8 +70,8 @@ typedef struct extent_tree_node { typedef struct rt_extent_tree_node { avlnode_t avl_node; - xfs_rtblock_t rt_startblock; /* starting realtime block */ - xfs_extlen_t rt_blockcount; /* number of blocks in extent */ + xfs_rtblock_t rt_startrtx; /* starting rt extent number */ + xfs_extlen_t rt_rtxlen; /* number of rt extents */ extent_state_t rt_state; /* see state flags below */ #if 0
@@ -157,11 +157,8 @@ int add_dup_extent(xfs_agnumber_t agno, xfs_agblock_t startblock, xfs_extlen_t blockcount);int search_dup_extent (xfs_agnumber_t agno,xfs_agblock_t start_agbno,xfs_agblock_t end_agbno); -void add_rt_dup_extent (xfs_rtblock_t startblock, -xfs_extlen_t blockcount); - -int search_rt_dup_extent (xfs_mount_t *mp, -xfs_rtblock_t bno); +void add_rt_dup_extent (xfs_rtblock_t startrtx,xfs_extlen_t rtxlen); +int search_rt_dup_extent (struct xfs_mount *mp,xfs_rtblock_t rtx);/* * extent/tree recyling and deletion routines
repair/incore_ext.c +39 -35
148 lines not on this page · raw message
repair/phase4.c +6 -6
@@ -250,7 +250,7 @@ void phase4(xfs_mount_t *mp) {ino_tree_node_t *irec; -xfs_rtblock_t bno; +xfs_rtblock_t rtx;xfs_rtblock_t rt_start;xfs_extlen_t rt_len;xfs_agnumber_t i;
@@ -331,14 +331,14 @@ phase4(xfs_mount_t *mp) rt_start =0 ; rt_len =0 ; -for (bno =0 ; bno < mp->m_sb.sb_rextents; bno++) { - bstate =get_rtbmap (bno); +for (rtx =0 ; rtx < mp->m_sb.sb_rextents; rtx++) { + bstate =get_rtbmap (rtx);switch (bstate) {case XR_E_BAD_STATE :default :do_warn (_ ("unknown rt extent state, extent %" PRIu64"\n" ), - bno); + rtx); fallthrough;case XR_E_UNKNOWN :case XR_E_FREE1 :
@@ -360,14 +360,14 @@ phase4(xfs_mount_t *mp)break ;case XR_E_MULT :if (rt_start ==0 ) { - rt_start = bno; + rt_start = rtx; rt_len =1 ; }else if (rt_len ==XFS_MAX_BMBT_EXTLEN ) {/* * large extent case */ add_rt_dup_extent (rt_start, rt_len); - rt_start = bno; + rt_start = rtx; rt_len =1 ; }else rt_len++;
PATCH 4/13 libxfs: create a helper to compute leftovers of realtime extents
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port the inode item precommunt function to use a helper to compute the misalignment between a file extent (xfs_extlen_t) and a realtime extent.
--- libxfs/logitem.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
libxfs/logitem.c +2 -1
@@ -15,6 +15,7 @@#include "xfs_inode_fork.h" #include "xfs_inode.h" #include "xfs_trans.h" +#include "xfs_rtbitmap.h" struct kmem_cache *xfs_buf_item_cache;struct kmem_cache *xfs_ili_cache;/* inode log item cache */
@@ -213,7 +214,7 @@ xfs_inode_item_precommit(*/ if ((ip->i_diflags &XFS_DIFLAG_RTINHERIT ) && (ip->i_diflags &XFS_DIFLAG_EXTSZINHERIT ) && - (ip->i_extsize % ip->i_mount->m_sb.sb_rextsize) >0 ) { +xfs_extlen_to_rtxmod (ip->i_mount, ip->i_extsize) >0 ) { ip->i_diflags &= ~(XFS_DIFLAG_EXTSIZE |XFS_DIFLAG_EXTSZINHERIT ); ip->i_extsize =0 ;
PATCH 5/13 libxfs: use helpers to convert rt block numbers to rt extent numbers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Now that we have helpers to do unit conversions of rt block numbers to rt extent numbers, plug that into libxfs.
--- libxfs/trans.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
libxfs/trans.c +2 -1
@@ -19,6 +19,7 @@#include "xfs_sb.h" #include "xfs_defer.h" #include "xfs_trace.h" +#include "xfs_rtbitmap.h" static void xfs_trans_free_items (struct xfs_trans *tp); STATIC struct xfs_trans *xfs_trans_dup(struct xfs_trans *tp);
@@ -1131,7 +1132,7 @@ libxfs_trans_alloc_inode(int error; error =libxfs_trans_alloc (mp, resv, dblocks, - rblocks / mp->m_sb.sb_rextsize, +xfs_rtb_to_rtx (mp, rblocks), force ?XFS_TRANS_RESERVE :0 , &tp);if (error)return error;
PATCH 6/13 xfs_repair: convert utility to use new rt extent helpers and types
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Convert the repair program to use the new realtime extent types and helper functions instead of open-coding them.
--- db/check.c | 2 +- repair/agheader.h | 2 +- repair/dinode.c | 21 ++++++++++++--------- repair/incore.c | 4 ++-- repair/incore.h | 12 ++++++------ repair/incore_ext.c | 14 +++++++------- repair/phase4.c | 6 +++--- repair/rt.c | 4 ++-- repair/scan.c | 2 +- 9 files changed, 35 insertions(+), 32 deletions(-)
db/check.c +1 -1
@@ -3594,7 +3594,7 @@ process_rtbitmap(int bitsperblock;xfs_fileoff_t bmbno;xfs_fsblock_t bno; -xfs_rtblock_t extno; +xfs_rtxnum_t extno;int len;int log;int offs;
repair/agheader.h +1 -1
@@ -11,7 +11,7 @@ typedef struct fs_geometry {uint32_t sb_blocksize;/* blocksize (bytes) */ xfs_rfsblock_t sb_dblocks;/* # data blocks */ xfs_rfsblock_t sb_rblocks;/* # realtime blocks */ -xfs_rtblock_t sb_rextents;/* # realtime extents */ +xfs_rtxnum_t sb_rextents;/* # realtime extents */ xfs_fsblock_t sb_logstart;/* starting log block # */ xfs_agblock_t sb_rextsize;/* realtime extent size (blocks )*/ xfs_agblock_t sb_agblocks;/* # of blocks per ag */
repair/dinode.c +12 -9
@@ -193,13 +193,13 @@ process_rt_rec_dups(xfs_ino_t ino, struct xfs_bmbt_irec *irec) { - xfs_fsblock_t b; -xfs_rtblock_t ext; + xfs_rtblock_t b; +xfs_rtxnum_t ext; -for (b =rounddown (irec->br_startblock, mp->m_sb.sb_rextsize); +for (b =xfs_rtb_rounddown_rtx (mp, irec->br_startblock); b < irec->br_startblock + irec->br_blockcount; b += mp->m_sb.sb_rextsize) { - ext = (xfs_rtblock_t ) b / mp->m_sb.sb_rextsize; + ext =xfs_rtb_to_rtx (mp, b);if (search_rt_dup_extent (mp, ext)) { do_warn( _("data fork in rt ino %" PRIu64 " claims dup rt extent,"
@@ -222,14 +222,17 @@ process_rt_rec_state( struct xfs_bmbt_irec *irec) {xfs_fsblock_t b = irec->br_startblock; -xfs_rtblock_t ext; +xfs_rtxnum_t ext;int state; do { - ext = (xfs_rtblock_t)b / mp->m_sb.sb_rextsize; + xfs_extlen_t mod; + + ext = xfs_rtb_to_rtx(mp, b); state = get_rtbmap(ext); - if ((b % mp->m_sb.sb_rextsize) != 0) { + mod = xfs_rtb_to_rtxoff(mp, b); + if (mod) { /* * We are midway through a partially written extent. * If we don't find the state that gets set in the
@@ -240,7 +243,7 @@ process_rt_rec_state(do_error (_ ("data fork in rt inode %" PRIu64" found invalid rt extent %" PRIu64" state %d at rt block %" PRIu64"\n" ), ino, ext, state, b); - b =roundup (b, mp->m_sb.sb_rextsize); + b =xfs_rtb_roundup_rtx (mp, b);continue ; }
@@ -2232,7 +2235,7 @@ validate_extsize(*/ if ((flags &XFS_DIFLAG_EXTSZINHERIT ) && (flags &XFS_DIFLAG_RTINHERIT ) && - value % mp->m_sb.sb_rextsize >0 ) +xfs_extlen_to_rtxmod (mp, value) >0 ) misaligned = true;/*
repair/incore.c +2 -2
@@ -178,7 +178,7 @@ static size_t rt_bmap_size;*/ int get_rtbmap ( -xfs_rtblock_t rtx) +xfs_rtxnum_t rtx) {return (*(rt_bmap + rtx /XR_BB_NUM ) >> ((rtx %XR_BB_NUM ) *XR_BB )) &XR_BB_MASK ;
@@ -186,7 +186,7 @@ get_rtbmap(void set_rtbmap ( -xfs_rtblock_t rtx, +xfs_rtxnum_t rtx,int state) { *(rt_bmap + rtx / XR_BB_NUM) =
repair/incore.h +6 -6
@@ -28,8 +28,8 @@ void set_bmap_ext(xfs_agnumber_t agno, xfs_agblock_t agbno,int get_bmap_ext (xfs_agnumber_t agno,xfs_agblock_t agbno,xfs_agblock_t maxbno,xfs_extlen_t *blen); -void set_rtbmap (xfs_rtblock_t rtx,int state); -int get_rtbmap (xfs_rtblock_t rtx); +void set_rtbmap (xfs_rtxnum_t rtx,int state); +int get_rtbmap (xfs_rtxnum_t rtx);static inline void set_bmap (xfs_agnumber_t agno,xfs_agblock_t agbno,int state)
@@ -70,8 +70,8 @@ typedef struct extent_tree_node { typedef struct rt_extent_tree_node { avlnode_t avl_node; - xfs_rtblock_t rt_startrtx; /* starting rt extent number */ - xfs_extlen_t rt_rtxlen; /* number of rt extents */ + xfs_rtxnum_t rt_startrtx; /* starting rt extent number */ + xfs_rtxlen_t rt_rtxlen; /* number of rt extents */ extent_state_t rt_state; /* see state flags below */ #if 0
@@ -157,8 +157,8 @@ int add_dup_extent(xfs_agnumber_t agno, xfs_agblock_t startblock, xfs_extlen_t blockcount);int search_dup_extent (xfs_agnumber_t agno,xfs_agblock_t start_agbno,xfs_agblock_t end_agbno); -void add_rt_dup_extent (xfs_rtblock_t startrtx,xfs_extlen_t rtxlen); -int search_rt_dup_extent (struct xfs_mount *mp,xfs_rtblock_t rtx); +void add_rt_dup_extent (xfs_rtxnum_t startrtx,xfs_rtxlen_t rtxlen); +int search_rt_dup_extent (struct xfs_mount *mp,xfs_rtxnum_t rtx);/* * extent/tree recyling and deletion routines
repair/incore_ext.c +7 -7
@@ -533,8 +533,8 @@ static avlops_t avl_extent_tree_ops = {*/ static rt_extent_tree_node_t *mk_rt_extent_tree_nodes ( -xfs_rtblock_t new_startrtx, -xfs_extlen_t new_rtxlen, +xfs_rtxnum_t new_startrtx, +xfs_rtxlen_t new_rtxlen,extent_state_t new_state) {struct rt_extent_tree_node *new;
@@ -603,12 +603,12 @@ free_rt_dup_extent_tree(xfs_mount_t *mp)*/ void add_rt_dup_extent ( -xfs_rtblock_t startrtx, -xfs_extlen_t rtxlen) +xfs_rtxnum_t startrtx, +xfs_rtxlen_t rtxlen) {struct rt_extent_tree_node *first, *last, *ext, *next_ext; -xfs_rtblock_t new_startrtx; -xfs_extlen_t new_rtxlen; +xfs_rtxnum_t new_startrtx; +xfs_rtxlen_t new_rtxlen;pthread_mutex_lock (&rt_ext_tree_lock); avl64_findranges(rt_ext_tree_ptr, startrtx - 1,
@@ -692,7 +692,7 @@ add_rt_dup_extent(int search_rt_dup_extent (struct xfs_mount *mp, -xfs_rtblock_t rtx) +xfs_rtxnum_t rtx) {int ret;
repair/phase4.c +3 -3
@@ -250,9 +250,9 @@ void phase4(xfs_mount_t *mp) {ino_tree_node_t *irec; -xfs_rtblock_t rtx; -xfs_rtblock_t rt_start; -xfs_extlen_t rt_len; +xfs_rtxnum_t rtx; +xfs_rtxnum_t rt_start; +xfs_rtxlen_t rt_len;xfs_agnumber_t i;xfs_agblock_t j;xfs_agblock_t ag_end;
repair/rt.c +2 -2
@@ -48,8 +48,8 @@ generate_rtinfo(xfs_mount_t *mp, xfs_rtword_t *words, xfs_suminfo_t *sumcompute) { -xfs_rtblock_t extno; -xfs_rtblock_t start_ext; +xfs_rtxnum_t extno; +xfs_rtxnum_t start_ext;int bitsperblock;int bmbno;xfs_rtword_t freebit;
repair/scan.c +1 -1
@@ -402,7 +402,7 @@ _("bad state %d, inode %" PRIu64 " bmap block 0x%" PRIx64 "\n"), XFS_FSB_TO_AGBNO(mp, bno) + 1))return (1 ); }else { -xfs_rtblock_t ext = bno / mp->m_sb.sb_rextsize; +xfs_rtxnum_t ext =xfs_rtb_to_rtx (mp, bno);if (search_rt_dup_extent (mp, ext))return 1 ;
PATCH 7/13 mkfs: convert utility to use new rt extent helpers and types
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Convert the repair program to use the new realtime extent types and helper functions instead of open-coding them.
--- mkfs/proto.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
mkfs/proto.c +28 -13
@@ -20,6 +20,7 @@ static int newfile(xfs_trans_t *tp, xfs_inode_t *ip, int symlink, int logit, char *buf, int len);static char *newregfile (char **pp,int *len);static void rtinit (xfs_mount_t *mp); +static void rtfreesp_init (struct xfs_mount *mp);static long filesize (int fd);static int slashes_are_spaces;
@@ -652,7 +653,6 @@ rtinit( xfs_mount_t *mp) {xfs_fileoff_t bno; -xfs_fileoff_t ebno;xfs_bmbt_irec_t *ep;int error;int i;
@@ -770,19 +770,34 @@ rtinit(fail (_ ("Block allocation of the realtime summary inode failed" ), error); -/* -* Free the whole area using transactions. -* Do one transaction per bitmap block. -*/ -for (bno =0 ; bno < mp->m_sb.sb_rextents; bno = ebno) { - i = -libxfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate, +rtfreesp_init (mp); +} + +/* +* Free the whole realtime area using transactions. +* Do one transaction per bitmap block. +*/ +static void +rtfreesp_init ( +struct xfs_mount *mp) +{ +struct xfs_trans *tp; +xfs_rtxnum_t rtx; +xfs_rtxnum_t ertx; +int error; + +for (rtx =0 ; rtx < mp->m_sb.sb_rextents; rtx = ertx) { + error = -libxfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate,0 ,0 ,0 , &tp); -if (i) -res_failed (i); -libxfs_trans_ijoin (tp, rbmip,0 ); - ebno =XFS_RTMIN (mp->m_sb.sb_rextents, - bno +NBBY * mp->m_sb.sb_blocksize); - error = -libxfs_rtfree_extent (tp, bno, (xfs_extlen_t)(ebno-bno)); +if (error) +res_failed (error); + +libxfs_trans_ijoin (tp, mp->m_rbmip,0 ); + ertx =XFS_RTMIN (mp->m_sb.sb_rextents, + rtx +NBBY * mp->m_sb.sb_blocksize); + + error = -libxfs_rtfree_extent (tp, rtx, + (xfs_rtxlen_t)(ertx - rtx));if (error) {fail (_ ("Error initializing the realtime space" ), error);
PATCH 8/13 xfs_{db,repair}: convert open-coded xfs_rtword_t pointer accesses to helper
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
There are a bunch of places in xfs_db and xfs_repair where we use open-coded logic to find a pointer to an xfs_rtword_t within a rt bitmap buffer. Convert all that to helper functions for better type safety.
--- db/check.c | 12 +++++++++--- repair/phase6.c | 12 ++++++++++-- 2 files changed, 19 insertions(+), 5 deletions(-)
db/check.c +9 -3
@@ -3608,8 +3608,11 @@ process_rtbitmap( bitsperblock = mp->m_sb.sb_blocksize *NBBY ; bit = extno = prevbit = start_bmbno = start_bit =0 ; bmbno =NULLFILEOFF ; -while ((bmbno =blkmap_next_off (blkmap, bmbno, &t)) != -NULLFILEOFF ) { +while ((bmbno =blkmap_next_off (blkmap, bmbno, &t)) !=NULLFILEOFF ) { +struct xfs_rtalloc_args args = { + .mp = mp, + }; + bno =blkmap_get (blkmap, bmbno);if (bno ==NULLFSBLOCK ) {if (!sflag)
@@ -3622,7 +3625,7 @@ process_rtbitmap(push_cur ();set_cur (&typtab[TYP_RTBITMAP ],XFS_FSB_TO_DADDR (mp, bno), blkbb,DB_RING_IGN ,NULL ); -if ((words = iocur_top->data) ==NULL ) { +if (!iocur_top->bp) {if (!sflag) dbprintf(_("can't read block %lld for rtbitmap " "inode\n"),
@@ -3631,6 +3634,9 @@ process_rtbitmap(pop_cur ();continue ; } + + args.rbmbp = iocur_top->bp; + words = (xfs_rtword_t *)xfs_rbmblock_wordptr (&args,0 );for (bit =0 ; bit < bitsperblock && extno < mp->m_sb.sb_rextents; bit++, extno++) {
repair/phase6.c +10 -2
@@ -593,6 +593,12 @@ fill_rbmino(xfs_mount_t *mp) }while (bno < mp->m_sb.sb_rbmblocks) { +struct xfs_rtalloc_args args = { + .mp = mp, + .tp = tp, + }; +union xfs_rtword_raw *ondisk; +/* * fill the file one block at a time */
@@ -618,11 +624,13 @@ _("can't access block %" PRIu64 " (fsbno %" PRIu64 ") of realtime bitmap inode %return (1 ); } -memmove (bp->b_addr, bmp, mp->m_sb.sb_blocksize); + args.rbmbp = bp; + ondisk =xfs_rbmblock_wordptr (&args,0 ); +memcpy (ondisk, bmp, mp->m_sb.sb_blocksize);libxfs_trans_log_buf (tp, bp,0 , mp->m_sb.sb_blocksize -1 ); - bmp = (xfs_rtword_t *)((intptr_t ) bmp + mp->m_sb.sb_blocksize); + bmp += mp->m_blockwsize; bno++; }
PATCH 9/13 xfs_repair: convert helpers for rtbitmap block/wordcount computations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port xfs_repair to use the new helper functions that compute the number of blocks or words necessary to store the rt bitmap.
--- libxfs/libxfs_api_defs.h | 2 ++ repair/rt.c | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-)
libxfs/libxfs_api_defs.h +2 -0
@@ -175,6 +175,8 @@#define xfs_rmap_lookup_le_range libxfs_rmap_lookup_le_range#define xfs_rmap_query_range libxfs_rmap_query_range +#define xfs_rtbitmap_wordcount libxfs_rtbitmap_wordcount +#define xfs_rtfree_extent libxfs_rtfree_extent#define xfs_rtfree_blocks libxfs_rtfree_blocks#define xfs_sb_from_disk libxfs_sb_from_disk
repair/rt.c +5 -5
@@ -19,6 +19,8 @@void rtinit (xfs_mount_t *mp) { +unsigned long long wordcnt; +if (mp->m_sb.sb_rblocks ==0 )return ;
@@ -26,11 +28,9 @@ rtinit(xfs_mount_t *mp)* realtime init -- blockmap initialization is * handled by incore_init() */ -/* -sumfile = calloc(mp->m_rsumsize, 1); -*/ -if ((btmcompute =calloc (mp->m_sb.sb_rbmblocks * - mp->m_sb.sb_blocksize,1 )) ==NULL ) + wordcnt =libxfs_rtbitmap_wordcount (mp, mp->m_sb.sb_rextents); + btmcompute =calloc (wordcnt,sizeof (xfs_rtword_t)); +if (!btmcompute)do_error (_ ("couldn't allocate memory for incore realtime bitmap.\n" ));
PATCH 10/13 xfs_{db,repair}: use accessor functions for bitmap words
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port xfs_db and xfs_repair to use get and set functions for rtbitmap words so that we can redefine the ondisk format with a specific endianness. Note that this requires the definition of a distinct type for ondisk rtbitmap words so that the compiler can perform proper typechecking as we go back and forth.
In the upcoming rtgroups feature, we're going to fix the problem that rtwords are written in host endian order, which means we'll need the distinct rtword/rtword_raw types.
--- db/check.c | 13 ++++++++++++- libxfs/libxfs_api_defs.h | 2 ++ repair/globals.c | 2 +- repair/globals.h | 2 +- repair/phase6.c | 2 +- repair/rt.c | 22 ++++++++++++++++------ repair/rt.h | 6 ++---- 7 files changed, 35 insertions(+), 14 deletions(-)
db/check.c +12 -1
@@ -3606,12 +3606,20 @@ process_rtbitmap(xfs_rtword_t *words; bitsperblock = mp->m_sb.sb_blocksize *NBBY ; + words =malloc (mp->m_blockwsize <<XFS_WORDLOG ); +if (!words) { +dbprintf (_ ("could not allocate rtwords buffer\n" )); + error++; +return ; + } bit = extno = prevbit = start_bmbno = start_bit =0 ; bmbno =NULLFILEOFF ;while ((bmbno =blkmap_next_off (blkmap, bmbno, &t)) !=NULLFILEOFF ) {struct xfs_rtalloc_args args = { .mp = mp, }; +xfs_rtword_t *incore = words; +unsigned int i; bno =blkmap_get (blkmap, bmbno);if (bno ==NULLFSBLOCK ) {
@@ -3636,7 +3644,9 @@ process_rtbitmap( } args.rbmbp = iocur_top->bp; - words = (xfs_rtword_t *)xfs_rbmblock_wordptr (&args,0 ); +for (i =0 ; i < mp->m_blockwsize; i++, incore++) + *incore =libxfs_rtbitmap_getword (&args, i); +for (bit =0 ; bit < bitsperblock && extno < mp->m_sb.sb_rextents; bit++, extno++) {
@@ -3670,6 +3680,7 @@ process_rtbitmap( offs =xfs_rtsumoffs (mp, log, start_bmbno); sumcompute[offs]++; } +free (words); } static void
libxfs/libxfs_api_defs.h +2 -0
@@ -175,6 +175,8 @@#define xfs_rmap_lookup_le_range libxfs_rmap_lookup_le_range#define xfs_rmap_query_range libxfs_rmap_query_range +#define xfs_rtbitmap_getword libxfs_rtbitmap_getword +#define xfs_rtbitmap_setword libxfs_rtbitmap_setword#define xfs_rtbitmap_wordcount libxfs_rtbitmap_wordcount#define xfs_rtfree_extent libxfs_rtfree_extent
repair/globals.c +1 -1
@@ -86,7 +86,7 @@ int64_t fs_max_file_offset;/* realtime info */ -xfs_rtword_t *btmcompute; +union xfs_rtword_raw *btmcompute;xfs_suminfo_t *sumcompute;/* inode tree records have full or partial backptr fields ? */
repair/globals.h +1 -1
@@ -127,7 +127,7 @@ extern int64_t fs_max_file_offset;/* realtime info */ -extern xfs_rtword_t *btmcompute; +extern union xfs_rtword_raw *btmcompute;extern xfs_suminfo_t *sumcompute;/* inode tree records have full or partial backptr fields ? */
repair/phase6.c +1 -1
@@ -572,7 +572,7 @@ fill_rbmino(xfs_mount_t *mp)struct xfs_buf *bp;xfs_trans_t *tp;xfs_inode_t *ip; -xfs_rtword_t *bmp; +union xfs_rtword_raw *bmp;int nmap;int error;xfs_fileoff_t bno;
repair/rt.c +16 -6
@@ -29,7 +29,7 @@ rtinit(xfs_mount_t *mp)* handled by incore_init() */ wordcnt =libxfs_rtbitmap_wordcount (mp, mp->m_sb.sb_rextents); - btmcompute =calloc (wordcnt,sizeof (xfs_rtword_t)); + btmcompute =calloc (wordcnt,sizeof (union xfs_rtword_raw ));if (!btmcompute)do_error (_ ("couldn't allocate memory for incore realtime bitmap.\n" ));
@@ -39,14 +39,24 @@ rtinit(xfs_mount_t *mp) _("couldn't allocate memory for incore realtime summary info.\n")); } +static inline void +set_rtword ( +struct xfs_mount *mp, +union xfs_rtword_raw *word, +xfs_rtword_t value) +{ + word->old = value; +} +/* * generate the real-time bitmap and summary info based on the * incore realtime extent map. */ int -generate_rtinfo (xfs_mount_t *mp, -xfs_rtword_t *words, -xfs_suminfo_t *sumcompute) +generate_rtinfo ( +struct xfs_mount *mp, +union xfs_rtword_raw *words, +xfs_suminfo_t *sumcompute) {xfs_rtxnum_t extno;xfs_rtxnum_t start_ext;
@@ -75,7 +85,7 @@ generate_rtinfo(xfs_mount_t *mp,*/ while (extno < mp->m_sb.sb_rextents) { freebit =1 ; - *words =0 ; +set_rtword (mp, words,0 ); bits =0 ;for (i =0 ; i <sizeof (xfs_rtword_t) *NBBY && extno < mp->m_sb.sb_rextents; i++, extno++) {
@@ -98,7 +108,7 @@ generate_rtinfo(xfs_mount_t *mp, freebit <<=1 ; } - *words = bits; +set_rtword (mp, words, bits); words++;if (extno % bitsperblock ==0 )
repair/rt.h +2 -4
@@ -11,10 +11,8 @@ struct blkmap;void rtinit (xfs_mount_t *mp); -int -generate_rtinfo (xfs_mount_t *mp, -xfs_rtword_t *words, -xfs_suminfo_t *sumcompute); +int generate_rtinfo (struct xfs_mount *mp,union xfs_rtword_raw *words, +xfs_suminfo_t *sumcompute);void check_rtbitmap (struct xfs_mount *mp);void check_rtsummary (struct xfs_mount *mp);
PATCH 11/13 xfs_{db,repair}: use helpers for rtsummary block/wordcount computations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port xfs_db and xfs_repair to use the new helper functions that compute the number of blocks or words necessary to store the rt summary file.
--- db/check.c | 8 ++++++-- libxfs/init.c | 8 ++++---- libxfs/libxfs_api_defs.h | 2 ++ repair/rt.c | 5 ++++- 4 files changed, 16 insertions(+), 7 deletions(-)
db/check.c +6 -2
@@ -1944,10 +1944,14 @@ init( inodata[c] =xcalloc (inodata_hash_size,sizeof (**inodata)); }if (rt) { +unsigned long long words; + dbmap[c] =xcalloc (mp->m_sb.sb_rblocks,sizeof (**dbmap)); inomap[c] =xcalloc (mp->m_sb.sb_rblocks,sizeof (**inomap)); - sumfile =xcalloc (mp->m_rsumsize,1 ); - sumcompute =xcalloc (mp->m_rsumsize,1 ); + words =libxfs_rtsummary_wordcount (mp, mp->m_rsumlevels, + mp->m_sb.sb_rbmblocks); + sumfile =xcalloc (words,sizeof (xfs_suminfo_t)); + sumcompute =xcalloc (words,sizeof (xfs_suminfo_t)); } nflag = sflag = tflag = verbose = optind =0 ;while ((c =getopt (argc, argv,"b:i:npstv" )) !=EOF ) {
libxfs/init.c +4 -4
@@ -284,6 +284,7 @@ rtmount_init( {struct xfs_buf *bp;/* buffer for last block of subvolume */ xfs_daddr_t d;/* address of last block of subvolume */ +unsigned int rsumblocks;int error;if (mp->m_sb.sb_rblocks ==0 )
@@ -309,10 +310,9 @@ rtmount_init(return -1 ; } mp->m_rsumlevels = mp->m_sb.sb_rextslog +1 ; - mp->m_rsumsize = - (uint )sizeof (xfs_suminfo_t) * mp->m_rsumlevels * - mp->m_sb.sb_rbmblocks; - mp->m_rsumsize =roundup (mp->m_rsumsize, mp->m_sb.sb_blocksize); + rsumblocks =xfs_rtsummary_blockcount (mp, mp->m_rsumlevels, + mp->m_sb.sb_rbmblocks); + mp->m_rsumsize =XFS_FSB_TO_B (mp, rsumblocks); mp->m_rbmip = mp->m_rsumip =NULL ;/*
libxfs/libxfs_api_defs.h +2 -0
@@ -179,6 +179,8 @@#define xfs_rtbitmap_setword libxfs_rtbitmap_setword#define xfs_rtbitmap_wordcount libxfs_rtbitmap_wordcount +#define xfs_rtsummary_wordcount libxfs_rtsummary_wordcount +#define xfs_rtfree_extent libxfs_rtfree_extent#define xfs_rtfree_blocks libxfs_rtfree_blocks#define xfs_sb_from_disk libxfs_sb_from_disk
repair/rt.c +4 -1
@@ -34,7 +34,10 @@ rtinit(xfs_mount_t *mp)do_error (_ ("couldn't allocate memory for incore realtime bitmap.\n" )); -if ((sumcompute =calloc (mp->m_rsumsize,1 )) ==NULL ) + wordcnt =libxfs_rtsummary_wordcount (mp, mp->m_rsumlevels, + mp->m_sb.sb_rbmblocks); + sumcompute =calloc (wordcnt,sizeof (xfs_suminfo_t)); +if (!sumcompute)do_error (_ ("couldn't allocate memory for incore realtime summary info.\n" )); }
PATCH 12/13 xfs_{db,repair}: use accessor functions for summary info words
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Port xfs_db and xfs_repair to use get and set functions for rtsummary words so that we can redefine the ondisk format with a specific endianness. Note that this requires the definition of a distinct type for ondisk summary info words so that the compiler can perform proper typechecking.
--- db/check.c | 58 +++++++++++++++++++++++++++++++++++----------- libxfs/libxfs_api_defs.h | 2 ++ repair/globals.c | 2 +- repair/globals.h | 2 +- repair/phase6.c | 14 +++++++++-- repair/rt.c | 19 ++++++++++++--- repair/rt.h | 2 +- 7 files changed, 75 insertions(+), 24 deletions(-)
db/check.c +44 -14
@@ -132,8 +132,8 @@ static unsigned sbversion;static int sbver_err;static int serious_error;static int sflag; -static xfs_suminfo_t *sumcompute; -static xfs_suminfo_t *sumfile; +static union xfs_suminfo_raw *sumcompute; +static union xfs_suminfo_raw *sumfile;static const char *typename[] = {"unknown" ,"agf" ,
@@ -1704,12 +1704,20 @@ check_set_rdbmap( } } +static inline xfs_suminfo_t +get_suminfo ( +struct xfs_mount *mp, +union xfs_suminfo_raw *raw) +{ +return raw->old; +} +static void check_summary (void ) {xfs_rfsblock_t bno; -xfs_suminfo_t *csp; -xfs_suminfo_t *fsp; +union xfs_suminfo_raw *csp; +union xfs_suminfo_raw *fsp;int log; csp = sumcompute;
@@ -1718,12 +1726,14 @@ check_summary(void)for (bno =0 ; bno < mp->m_sb.sb_rbmblocks; bno++, csp++, fsp++) { -if (*csp != *fsp) { +if (csp->old != fsp->old) {if (!sflag)dbprintf (_ ("rt summary mismatch, size %d " "block %llu, file: %d, " "computed: %d\n" ), - log, bno, *fsp, *csp); + log, bno, +get_suminfo (mp, fsp), +get_suminfo (mp, csp)); error++; } }
@@ -1950,8 +1960,8 @@ init( inomap[c] =xcalloc (mp->m_sb.sb_rblocks,sizeof (**inomap)); words =libxfs_rtsummary_wordcount (mp, mp->m_rsumlevels, mp->m_sb.sb_rbmblocks); - sumfile =xcalloc (words,sizeof (xfs_suminfo_t)); - sumcompute =xcalloc (words,sizeof (xfs_suminfo_t)); + sumfile =xcalloc (words,sizeof (union xfs_suminfo_raw )); + sumcompute =xcalloc (words,sizeof (union xfs_suminfo_raw )); } nflag = sflag = tflag = verbose = optind =0 ;while ((c =getopt (argc, argv,"b:i:npstv" )) !=EOF ) {
@@ -3590,6 +3600,17 @@ process_quota( } } +static inline void +inc_sumcount ( +struct xfs_mount *mp, +union xfs_suminfo_raw *info, +xfs_rtsumoff_t index) +{ +union xfs_suminfo_raw *p = info + index; + + p->old++; +} +static void process_rtbitmap (blkmap_t *blkmap)
@@ -3669,7 +3690,7 @@ process_rtbitmap( bitsperblock + (bit - start_bit); log =XFS_RTBLOCKLOG (len); offs =xfs_rtsumoffs (mp, log, start_bmbno); - sumcompute[offs]++; +inc_sumcount (mp, sumcompute, offs); prevbit =0 ; } }
@@ -3682,7 +3703,7 @@ process_rtbitmap( (bit - start_bit); log =XFS_RTBLOCKLOG (len); offs =xfs_rtsumoffs (mp, log, start_bmbno); - sumcompute[offs]++; +inc_sumcount (mp, sumcompute, offs); }free (words); }
@@ -3692,12 +3713,17 @@ process_rtsummary( blkmap_t *blkmap) {xfs_fsblock_t bno; -char *bytes; +union xfs_suminfo_raw *sfile = sumfile;xfs_fileoff_t sumbno;int t; sumbno =NULLFILEOFF ;while ((sumbno =blkmap_next_off (blkmap, sumbno, &t)) !=NULLFILEOFF ) { +struct xfs_rtalloc_args args = { + .mp = mp, + }; +union xfs_suminfo_raw *ondisk; + bno =blkmap_get (blkmap, sumbno);if (bno ==NULLFSBLOCK ) {if (!sflag)
@@ -3710,18 +3736,22 @@ process_rtsummary(push_cur ();set_cur (&typtab[TYP_RTSUMMARY ],XFS_FSB_TO_DADDR (mp, bno), blkbb,DB_RING_IGN ,NULL ); -if ((bytes = iocur_top->data) ==NULL ) { +if (!iocur_top->bp) {if (!sflag)dbprintf (_ ("can't read block %lld for rtsummary " "inode\n" ), (xfs_fileoff_t )sumbno); error++;pop_cur (); + sfile += mp->m_blockwsize;continue ; } -memcpy ((char *)sumfile + sumbno * mp->m_sb.sb_blocksize, bytes, - mp->m_sb.sb_blocksize); + + args.sumbp = iocur_top->bp; + ondisk =xfs_rsumblock_infoptr (&args,0 ); +memcpy (sfile, ondisk, mp->m_sb.sb_blocksize);pop_cur (); + sfile += mp->m_blockwsize; } }
libxfs/libxfs_api_defs.h +2 -0
@@ -179,6 +179,8 @@#define xfs_rtbitmap_setword libxfs_rtbitmap_setword#define xfs_rtbitmap_wordcount libxfs_rtbitmap_wordcount +#define xfs_suminfo_add libxfs_suminfo_add +#define xfs_suminfo_get libxfs_suminfo_get#define xfs_rtsummary_wordcount libxfs_rtsummary_wordcount#define xfs_rtfree_extent libxfs_rtfree_extent
repair/globals.c +1 -1
@@ -87,7 +87,7 @@ int64_t fs_max_file_offset;/* realtime info */ union xfs_rtword_raw *btmcompute; -xfs_suminfo_t *sumcompute; +union xfs_suminfo_raw *sumcompute;/* inode tree records have full or partial backptr fields ? */
repair/globals.h +1 -1
@@ -128,7 +128,7 @@ extern int64_t fs_max_file_offset;/* realtime info */ extern union xfs_rtword_raw *btmcompute; -extern xfs_suminfo_t *sumcompute; +extern union xfs_suminfo_raw *sumcompute;/* inode tree records have full or partial backptr fields ? */
repair/phase6.c +11 -3
@@ -648,7 +648,7 @@ fill_rsumino(xfs_mount_t *mp)struct xfs_buf *bp;xfs_trans_t *tp;xfs_inode_t *ip; -xfs_suminfo_t *smp; +union xfs_suminfo_raw *smp;int nmap;int error;xfs_fileoff_t bno;
@@ -671,6 +671,12 @@ fill_rsumino(xfs_mount_t *mp) }while (bno < end_bno) { +struct xfs_rtalloc_args args = { + .mp = mp, + .tp = tp, + }; +union xfs_suminfo_raw *ondisk; +/* * fill the file one block at a time */
@@ -697,11 +703,13 @@ _("can't access block %" PRIu64 " (fsbno %" PRIu64 ") of realtime summary inodereturn (1 ); } -memmove (bp->b_addr, smp, mp->m_sb.sb_blocksize); + args.sumbp = bp; + ondisk =xfs_rsumblock_infoptr (&args,0 ); +memcpy (ondisk, smp, mp->m_sb.sb_blocksize);libxfs_trans_log_buf (tp, bp,0 , mp->m_sb.sb_blocksize -1 ); - smp = (xfs_suminfo_t *)((intptr_t )smp + mp->m_sb.sb_blocksize); + smp += mp->m_blockwsize; bno++; }
repair/rt.c +15 -4
@@ -36,7 +36,7 @@ rtinit(xfs_mount_t *mp) wordcnt =libxfs_rtsummary_wordcount (mp, mp->m_rsumlevels, mp->m_sb.sb_rbmblocks); - sumcompute =calloc (wordcnt,sizeof (xfs_suminfo_t)); + sumcompute =calloc (wordcnt,sizeof (union xfs_suminfo_raw ));if (!sumcompute)do_error (_ ("couldn't allocate memory for incore realtime summary info.\n" ));
@@ -51,6 +51,17 @@ set_rtword( word->old = value; } +static inline void +inc_sumcount ( +struct xfs_mount *mp, +union xfs_suminfo_raw *info, +xfs_rtsumoff_t index) +{ +union xfs_suminfo_raw *p = info + index; + + p->old++; +} +/* * generate the real-time bitmap and summary info based on the * incore realtime extent map.
@@ -59,7 +70,7 @@ intgenerate_rtinfo ( struct xfs_mount *mp, union xfs_rtword_raw *words, - xfs_suminfo_t *sumcompute) + union xfs_suminfo_raw *sumcompute) { xfs_rtxnum_t extno;xfs_rtxnum_t start_ext;
@@ -105,7 +116,7 @@ generate_rtinfo( len = (int ) (extno - start_ext); log =XFS_RTBLOCKLOG (len); offs =xfs_rtsumoffs (mp, log, start_bmbno); - sumcompute[offs]++; +inc_sumcount (mp, sumcompute, offs); in_extent =0 ; }
@@ -121,7 +132,7 @@ generate_rtinfo( len = (int ) (extno - start_ext); log =XFS_RTBLOCKLOG (len); offs =xfs_rtsumoffs (mp, log, start_bmbno); - sumcompute[offs]++; +inc_sumcount (mp, sumcompute, offs); }if (mp->m_sb.sb_frextents != sb_frextents) {
repair/rt.h +1 -1
@@ -12,7 +12,7 @@ voidrtinit (xfs_mount_t *mp);int generate_rtinfo (struct xfs_mount *mp,union xfs_rtword_raw *words, -xfs_suminfo_t *sumcompute); +union xfs_suminfo_raw *sumcompute);void check_rtbitmap (struct xfs_mount *mp);void check_rtsummary (struct xfs_mount *mp);
PATCH 13/13 xfs_{db,repair}: use m_blockwsize instead of sb_blocksize for rt blocks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
In preparation to add block headers to rt bitmap and summary blocks, convert all the relevant calculations in the userspace tools to use the per-block word count instead of the raw blocksize. This is key to adding this support outside of libxfs.
--- db/check.c | 4 ++-- repair/phase6.c | 4 ++-- repair/rt.c | 9 +++++---- 3 files changed, 9 insertions(+), 8 deletions(-)
db/check.c +2 -2
@@ -3630,7 +3630,7 @@ process_rtbitmap(int t;xfs_rtword_t *words; - bitsperblock = mp->m_sb.sb_blocksize *NBBY ; + bitsperblock = mp->m_blockwsize <<XFS_NBWORDLOG ; words =malloc (mp->m_blockwsize <<XFS_WORDLOG );if (!words) {dbprintf (_ ("could not allocate rtwords buffer\n" ));
@@ -3749,7 +3749,7 @@ process_rtsummary( args.sumbp = iocur_top->bp; ondisk =xfs_rsumblock_infoptr (&args,0 ); -memcpy (sfile, ondisk, mp->m_sb.sb_blocksize); +memcpy (sfile, ondisk, mp->m_blockwsize <<XFS_WORDLOG );pop_cur (); sfile += mp->m_blockwsize; }
repair/phase6.c +2 -2
@@ -626,7 +626,7 @@ _("can't access block %" PRIu64 " (fsbno %" PRIu64 ") of realtime bitmap inode % args.rbmbp = bp; ondisk =xfs_rbmblock_wordptr (&args,0 ); -memcpy (ondisk, bmp, mp->m_sb.sb_blocksize); +memcpy (ondisk, bmp, mp->m_blockwsize <<XFS_WORDLOG );libxfs_trans_log_buf (tp, bp,0 , mp->m_sb.sb_blocksize -1 );
@@ -705,7 +705,7 @@ _("can't access block %" PRIu64 " (fsbno %" PRIu64 ") of realtime summary inode args.sumbp = bp; ondisk =xfs_rsumblock_infoptr (&args,0 ); -memcpy (ondisk, smp, mp->m_sb.sb_blocksize); +memcpy (ondisk, smp, mp->m_blockwsize <<XFS_WORDLOG );libxfs_trans_log_buf (tp, bp,0 , mp->m_sb.sb_blocksize -1 );
repair/rt.c +5 -4
@@ -25,8 +25,9 @@ rtinit(xfs_mount_t *mp)return ;/* -* realtime init -- blockmap initialization is -* handled by incore_init() +* Allocate buffers for formatting the collected rt free space +* information. The rtbitmap buffer must be large enough to compare +* against any unused bytes in the last block of the file. */ wordcnt =libxfs_rtbitmap_wordcount (mp, mp->m_sb.sb_rextents); btmcompute =calloc (wordcnt,sizeof (union xfs_rtword_raw ));
@@ -87,7 +88,7 @@ generate_rtinfo(ASSERT (mp->m_rbmip ==NULL ); - bitsperblock = mp->m_sb.sb_blocksize *NBBY ; + bitsperblock = mp->m_blockwsize <<XFS_NBWORDLOG ; extno = start_ext =0 ; bmbno = in_extent = start_bmbno =0 ;
@@ -199,7 +200,7 @@ check_rtfile_contents(break ; } -if (memcmp (bp->b_addr, buf, mp->m_sb.sb_blocksize)) +if (memcmp (bp->b_addr, buf, mp->m_blockwsize <<XFS_WORDLOG ))do_warn (_ ("discrepancy in %s at dblock 0x%llx\n" ), filename, (unsigned long long )bno);
PATCH 2/18 libxfs: sync with 6.8
In reply to Darrick J. WongHi all,
Synchronize libxfs with the kernel.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=libxfs-6.8-sync
---
Commits in this patchset:
* xfs: use xfs_defer_pending objects to recover intent items
* xfs: recreate work items when recovering intent items
* xfs: use xfs_defer_finish_one to finish recovered work items
* xfs: move ->iop_recover to xfs_defer_op_type
* xfs: hoist intent done flag setting to ->finish_item callsite
* xfs: hoist ->create_intent boilerplate to its callsite
* xfs: use xfs_defer_create_done for the relogging operation
* xfs: clean out XFS_LI_DIRTY setting boilerplate from ->iop_relog
* xfs: hoist xfs_trans_add_item calls to defer ops functions
* xfs: move ->iop_relog to struct xfs_defer_op_type
* xfs: make rextslog computation consistent with mkfs
* xfs: fix 32-bit truncation in xfs_compute_rextslog
* xfs: don't allow overly small or large realtime volumes
* xfs: elide ->create_done calls for unlogged deferred work
* xfs: don't append work items to logged xfs_defer_pending objects
* xfs: allow pausing of pending deferred work items
* xfs: remove __xfs_free_extent_later
* xfs: automatic freeing of freshly allocated unwritten space
* xfs: remove unused fields from struct xbtree_ifakeroot
* xfs: force small EFIs for reaping btree extents
* xfs: ensure logflagsp is initialized in xfs_bmap_del_extent_real
* xfs: update dir3 leaf block metadata after swap
* xfs: extract xfs_da_buf_copy() helper function
* xfs: move xfs_ondisk.h to libxfs/
* xfs: consolidate the xfs_attr_defer_* helpers
* xfs: store an ops pointer in struct xfs_defer_pending
* xfs: pass the defer ops instead of type to xfs_defer_start_recovery
* xfs: pass the defer ops directly to xfs_defer_add
* xfs: force all buffers to be written during btree bulk load
* xfs: set XBF_DONE on newly formatted btree block that are ready for writing
* xfs: read leaf blocks when computing keys for bulkloading into node blocks
* xfs: move btree bulkload record initialization to ->get_record implementations
* xfs: constrain dirty buffers while formatting a staged btree
* xfs: repair free space btrees
* xfs: repair inode btrees
* xfs: repair refcount btrees
* xfs: dont cast to char * for XFS_DFORK_*PTR macros
* xfs: set inode sick state flags when we zap either ondisk fork
* xfs: zap broken inode forks
* xfs: repair inode fork block mapping data structures
* xfs: create a ranged query function for refcount btrees
* xfs: create a new inode fork block unmap helper
* xfs: improve dquot iteration for scrub
* xfs: add lock protection when remove perag from radix tree
* xfs: fix perag leak when growfs fails
* xfs: remove the xfs_alloc_arg argument to xfs_bmap_btalloc_accounting
* xfs: also use xfs_bmap_btalloc_accounting for RT allocations
* xfs: return -ENOSPC from xfs_rtallocate_*
* xfs: indicate if xfs_bmap_adjacent changed ap->blkno
* xfs: move xfs_rtget_summary to xfs_rtbitmap.c
* xfs: split xfs_rtmodify_summary_int
* xfs: remove rt-wrappers from xfs_format.h
* xfs: remove XFS_RTMIN/XFS_RTMAX
* xfs: make if_data a void pointer
* xfs: return if_data from xfs_idata_realloc
* xfs: move the xfs_attr_sf_lookup tracepoint
* xfs: simplify xfs_attr_sf_findname
* xfs: remove xfs_attr_shortform_lookup
* xfs: use xfs_attr_sf_findname in xfs_attr_shortform_getvalue
* xfs: remove struct xfs_attr_shortform
* xfs: remove xfs_attr_sf_hdr_t
* xfs: turn the XFS_DA_OP_REPLACE checks in xfs_attr_shortform_addname into asserts
* xfs: fix a use after free in xfs_defer_finish_recovery
* xfs: use the op name in trace_xlog_intent_recovery_failed
* xfs: fix backwards logic in xfs_bmap_alloc_account
* xfs: reset XFS_ATTR_INCOMPLETE filter on node removal
* xfs: remove conditional building of rt geometry validator functions
--- db/attrshort.c | 35 ++- db/check.c | 12 + db/inode.c | 6 - db/metadump.c | 16 +- db/namei.c | 4 include/list.h | 14 + include/xfs_trace.h | 5 include/xfs_trans.h | 1 libxfs/defer_item.c | 16 +- libxfs/init.c | 6 + libxfs/libxfs_api_defs.h | 4 libxfs/libxfs_io.h | 11 + libxfs/libxfs_priv.h | 7 - libxfs/util.c | 2 libxfs/xfs_ag.c | 38 +++- libxfs/xfs_ag.h | 12 + libxfs/xfs_ag_resv.c | 2 libxfs/xfs_alloc.c | 116 ++++++++++- libxfs/xfs_alloc.h | 24 +- libxfs/xfs_alloc_btree.c | 13 + libxfs/xfs_attr.c | 131 ++++-------- libxfs/xfs_attr_leaf.c | 244 ++++++++--------------- libxfs/xfs_attr_leaf.h | 8 - libxfs/xfs_attr_sf.h | 24 +- libxfs/xfs_bmap.c | 201 ++++++++++++------- libxfs/xfs_bmap.h | 9 + libxfs/xfs_bmap_btree.c | 124 ++++++++++-- libxfs/xfs_bmap_btree.h | 5 libxfs/xfs_btree.c | 28 +++ libxfs/xfs_btree.h | 5 libxfs/xfs_btree_staging.c | 89 ++++++-- libxfs/xfs_btree_staging.h | 33 ++- libxfs/xfs_da_btree.c | 69 +++---- libxfs/xfs_da_btree.h | 2 libxfs/xfs_da_format.h | 31 ++- libxfs/xfs_defer.c | 452 +++++++++++++++++++++++++++++++++++-------- libxfs/xfs_defer.h | 59 ++++-- libxfs/xfs_dir2.c | 2 libxfs/xfs_dir2_block.c | 6 - libxfs/xfs_dir2_priv.h | 3 libxfs/xfs_dir2_sf.c | 91 +++------ libxfs/xfs_format.h | 19 -- libxfs/xfs_health.h | 10 + libxfs/xfs_ialloc.c | 36 ++- libxfs/xfs_ialloc.h | 3 libxfs/xfs_ialloc_btree.c | 2 libxfs/xfs_iext_tree.c | 59 +++--- libxfs/xfs_inode_fork.c | 78 ++++--- libxfs/xfs_inode_fork.h | 13 + libxfs/xfs_ondisk.h | 199 +++++++++++++++++++ libxfs/xfs_refcount.c | 57 +++++ libxfs/xfs_refcount.h | 12 + libxfs/xfs_refcount_btree.c | 15 + libxfs/xfs_rmap.c | 2 libxfs/xfs_rtbitmap.c | 134 ++++++------- libxfs/xfs_rtbitmap.h | 4 libxfs/xfs_sb.c | 20 ++ libxfs/xfs_sb.h | 2 libxfs/xfs_shared.h | 2 libxfs/xfs_symlink_remote.c | 12 - libxfs/xfs_types.h | 20 ++ mkfs/proto.c | 4 mkfs/xfs_mkfs.c | 8 + repair/agbtree.c | 57 ++++- repair/attr_repair.c | 48 ++--- repair/dinode.c | 23 +- repair/phase6.c | 9 - repair/rt.c | 6 - repair/sb.c | 6 - 69 files changed, 1861 insertions(+), 959 deletions(-) create mode 100644 libxfs/xfs_ondisk.h
PATCH 1/67 xfs: use xfs_defer_pending objects to recover intent items
From: Darrick J. Wong <[email protected]>
Source kernel commit: 03f7767c9f6120ac933378fdec3bfd78bf07bc11
One thing I never quite got around to doing is porting the log intent item recovery code to reconstruct the deferred pending work state. As a result, each intent item open codes xfs_defer_finish_one in its recovery method, because that's what the EFI code did before xfs_defer.c even existed.
This is a gross thing to have left unfixed -- if an EFI cannot proceed due to busy extents, we end up creating separate new EFIs for each unfinished work item, which is a change in behavior from what runtime would have done.
Worse yet, Long Li pointed out that there's a UAF in the recovery code. The ->commit_pass2 function adds the intent item to the AIL and drops the refcount. The one remaining refcount is now owned by the recovery mechanism (aka the log intent items in the AIL) with the intent of giving the refcount to the intent done item in the ->iop_recover function.
However, if something fails later in recovery, xlog_recover_finish will walk the recovered intent items in the AIL and release them. If the CIL hasn't been pushed before that point (which is possible since we don't force the log until later) then the intent done release will try to free its associated intent, which has already been freed.
This patch starts to address this mess by having the ->commit_pass2 functions recreate the xfs_defer_pending state. The next few patches will fix the recovery functions.
--- libxfs/xfs_defer.c | 105 ++++++++++++++++++++++++++++++++++++++-------------- libxfs/xfs_defer.h | 5 ++ 2 files changed, 82 insertions(+), 28 deletions(-)
libxfs/xfs_defer.c +77 -28
@@ -240,23 +240,53 @@ xfs_defer_create_intents(return ret; } -STATIC void +static inline void xfs_defer_pending_abort ( +struct xfs_mount *mp, +struct xfs_defer_pending *dfp) +{ +const struct xfs_defer_op_type *ops = defer_op_types[dfp->dfp_type]; + +trace_xfs_defer_pending_abort (mp, dfp); + +if (dfp->dfp_intent && !dfp->dfp_done) { + ops->abort_intent (dfp->dfp_intent); + dfp->dfp_intent =NULL ; + } +} + +static inline void +xfs_defer_pending_cancel_work ( +struct xfs_mount *mp, +struct xfs_defer_pending *dfp) +{ +const struct xfs_defer_op_type *ops = defer_op_types[dfp->dfp_type]; +struct list_head *pwi; +struct list_head *n; + +trace_xfs_defer_cancel_list (mp, dfp); + +list_del (&dfp->dfp_list); +list_for_each_safe (pwi, n, &dfp->dfp_work) { +list_del (pwi); + dfp->dfp_count--; +trace_xfs_defer_cancel_item (mp, dfp, pwi); + ops->cancel_item (pwi); + } +ASSERT (dfp->dfp_count ==0 ); +kmem_cache_free (xfs_defer_pending_cache, dfp); +} + +STATIC void +xfs_defer_pending_abort_list (struct xfs_mount *mp,struct list_head *dop_list) {struct xfs_defer_pending *dfp; -const struct xfs_defer_op_type *ops;/* Abort intent items that don't have a done item. */ -list_for_each_entry (dfp, dop_list, dfp_list) { - ops = defer_op_types[dfp->dfp_type]; -trace_xfs_defer_pending_abort (mp, dfp); -if (dfp->dfp_intent && !dfp->dfp_done) { - ops->abort_intent (dfp->dfp_intent); - dfp->dfp_intent =NULL ; - } - } +list_for_each_entry (dfp, dop_list, dfp_list) +xfs_defer_pending_abort (mp, dfp); }/* Abort all the intents that were committed. */
@@ -266,7 +296,7 @@ xfs_defer_trans_abort( struct list_head *dop_pending) {trace_xfs_defer_trans_abort (tp, _RET_IP_); -xfs_defer_pending_abort (tp->t_mountp, dop_pending); +xfs_defer_pending_abort_list (tp->t_mountp, dop_pending); }/*
@@ -384,27 +414,13 @@ xfs_defer_cancel_list( {struct xfs_defer_pending *dfp;struct xfs_defer_pending *pli; -struct list_head *pwi; -struct list_head *n; -const struct xfs_defer_op_type *ops;/* * Free the pending items. Caller should already have arranged * for the intent items to be released. */ -list_for_each_entry_safe (dfp, pli, dop_list, dfp_list) { - ops = defer_op_types[dfp->dfp_type]; -trace_xfs_defer_cancel_list (mp, dfp); -list_del (&dfp->dfp_list); -list_for_each_safe (pwi, n, &dfp->dfp_work) { -list_del (pwi); - dfp->dfp_count--; -trace_xfs_defer_cancel_item (mp, dfp, pwi); - ops->cancel_item (pwi); - } -ASSERT (dfp->dfp_count ==0 ); -kmem_cache_free (xfs_defer_pending_cache, dfp); - } +list_for_each_entry_safe (dfp, pli, dop_list, dfp_list) +xfs_defer_pending_cancel_work (mp, dfp); }/*
@@ -660,6 +676,39 @@ xfs_defer_add( dfp->dfp_count++; } +/* +* Create a pending deferred work item to replay the recovered intent item +* and add it to the list. +*/ +void +xfs_defer_start_recovery ( +struct xfs_log_item *lip, +enum xfs_defer_ops_type dfp_type, +struct list_head *r_dfops) +{ +struct xfs_defer_pending *dfp; + + dfp =kmem_cache_zalloc (xfs_defer_pending_cache, +GFP_NOFS | __GFP_NOFAIL); + dfp->dfp_type = dfp_type; + dfp->dfp_intent = lip; +INIT_LIST_HEAD (&dfp->dfp_work); +list_add_tail (&dfp->dfp_list, r_dfops); +} + +/* +* Cancel a deferred work item created to recover a log intent item. @dfp +* will be freed after this function returns. +*/ +void +xfs_defer_cancel_recovery ( +struct xfs_mount *mp, +struct xfs_defer_pending *dfp) +{ +xfs_defer_pending_abort (mp, dfp); +xfs_defer_pending_cancel_work (mp, dfp); +} +/* * Move deferred ops from one transaction to another and reset the source to * initial state. This is primarily used to carry state forward across
@@ -764,7 +813,7 @@ xfs_defer_ops_capture_abort( {unsigned short i; -xfs_defer_pending_abort (mp, &dfc->dfc_dfops); +xfs_defer_pending_abort_list (mp, &dfc->dfc_dfops);xfs_defer_cancel_list (mp, &dfc->dfc_dfops);for (i =0 ; i < dfc->dfc_held.dr_bufs; i++)
libxfs/xfs_defer.h +5 -0
@@ -125,6 +125,11 @@ void xfs_defer_ops_capture_abort(struct xfs_mount *mp, struct xfs_defer_capture *d);void xfs_defer_resources_rele (struct xfs_defer_resources *dres); +void xfs_defer_start_recovery (struct xfs_log_item *lip, +enum xfs_defer_ops_type dfp_type,struct list_head *r_dfops); +void xfs_defer_cancel_recovery (struct xfs_mount *mp, +struct xfs_defer_pending *dfp); +int __init xfs_defer_init_item_caches (void );void xfs_defer_destroy_item_caches (void );
PATCH 2/67 xfs: recreate work items when recovering intent items
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: e70fb328d5277297ea2d9169a3a046de6412d777
Recreate work items for each xfs_defer_pending object when we are recovering intent items.
--- libxfs/xfs_defer.c | 3 +-- libxfs/xfs_defer.h | 9 +++++++++ 2 files changed, 10 insertions(+), 2 deletions(-)
libxfs/xfs_defer.c +1 -2
@@ -671,9 +671,8 @@ xfs_defer_add(list_add_tail (&dfp->dfp_list, &tp->t_dfops); } -list_add_tail (li, &dfp->dfp_work); +xfs_defer_add_item (dfp, li);trace_xfs_defer_add_item (tp->t_mountp, dfp, li); - dfp->dfp_count++; }/*
libxfs/xfs_defer.h +9 -0
@@ -130,6 +130,15 @@ void xfs_defer_start_recovery(struct xfs_log_item *lip,void xfs_defer_cancel_recovery (struct xfs_mount *mp,struct xfs_defer_pending *dfp); +static inline void +xfs_defer_add_item ( +struct xfs_defer_pending *dfp, +struct list_head *work) +{ +list_add_tail (work, &dfp->dfp_work); + dfp->dfp_count++; +} +int __init xfs_defer_init_item_caches (void );void xfs_defer_destroy_item_caches (void );
PATCH 3/67 xfs: use xfs_defer_finish_one to finish recovered work items
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: e5f1a5146ec35f3ed5d7f5ac7807a10c0062b6b8
Get rid of the open-coded calls to xfs_defer_finish_one. This also means that the recovery transaction takes care of cleaning up the dfp, and we have solved (I hope) all the ownership issues in recovery.
--- libxfs/xfs_defer.c | 2 +- libxfs/xfs_defer.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
libxfs/xfs_defer.c +1 -1
@@ -479,7 +479,7 @@ xfs_defer_relog(* Log an intent-done item for the first pending intent, and finish the work * items. */ -static int +int xfs_defer_finish_one (struct xfs_trans *tp,struct xfs_defer_pending *dfp)
libxfs/xfs_defer.h +1 -0
@@ -41,6 +41,7 @@ void xfs_defer_add(struct xfs_trans *tp, enum xfs_defer_ops_type type, struct list_head *h);int xfs_defer_finish_noroll (struct xfs_trans **tp);int xfs_defer_finish (struct xfs_trans **tp); +int xfs_defer_finish_one (struct xfs_trans *tp,struct xfs_defer_pending *dfp);void xfs_defer_cancel (struct xfs_trans *);void xfs_defer_move (struct xfs_trans *dtp,struct xfs_trans *stp);
PATCH 4/67 xfs: move ->iop_recover to xfs_defer_op_type
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: db7ccc0bac2add5a41b66578e376b49328fc99d0
Finish off the series by moving the intent item recovery function pointer to the xfs_defer_op_type struct, since this is really a deferred work function now.
--- include/xfs_trace.h | 2 ++ libxfs/xfs_defer.c | 17 +++++++++++++++++ libxfs/xfs_defer.h | 4 ++++ 3 files changed, 23 insertions(+)
include/xfs_trace.h +2 -0
@@ -336,4 +336,6 @@#define trace_xfs_fs_mark_healthy (a,b) ((void) 0) +#define trace_xlog_intent_recovery_failed (...) ((void) 0) +#endif /* __TRACE_H__ */
libxfs/xfs_defer.c +17 -0
@@ -708,6 +708,23 @@ xfs_defer_cancel_recovery(xfs_defer_pending_cancel_work (mp, dfp); } +/* Replay the deferred work item created from a recovered log intent item. */ +int +xfs_defer_finish_recovery ( +struct xfs_mount *mp, +struct xfs_defer_pending *dfp, +struct list_head *capture_list) +{ +const struct xfs_defer_op_type *ops = defer_op_types[dfp->dfp_type]; +int error; + + error = ops->recover_work (dfp, capture_list); +if (error) +trace_xlog_intent_recovery_failed (mp, error, + ops->recover_work); +return error; +} +/* * Move deferred ops from one transaction to another and reset the source to * initial state. This is primarily used to carry state forward across
libxfs/xfs_defer.h +4 -0
@@ -57,6 +57,8 @@ struct xfs_defer_op_type {void (*finish_cleanup)(struct xfs_trans *tp,struct xfs_btree_cur *state,int error);void (*cancel_item)(struct list_head *item); +int (*recover_work)(struct xfs_defer_pending *dfp, +struct list_head *capture_list);unsigned int max_items; };
@@ -130,6 +132,8 @@ void xfs_defer_start_recovery(struct xfs_log_item *lip, enum xfs_defer_ops_type dfp_type, struct list_head *r_dfops);void xfs_defer_cancel_recovery (struct xfs_mount *mp,struct xfs_defer_pending *dfp); +int xfs_defer_finish_recovery (struct xfs_mount *mp, +struct xfs_defer_pending *dfp,struct list_head *capture_list);static inline void xfs_defer_add_item(
PATCH 5/67 xfs: hoist intent done flag setting to ->finish_item callsite
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 3dd75c8db1c1675a26d3e228bab349c1fc065867
Each log intent item's ->finish_item call chain inevitably includes some code to set the dirty flag of the transaction. If there's an associated log intent done item, it also sets the item's dirty flag and the transaction's INTENT_DONE flag. This is repeated throughout the codebase.
Reduce the LOC by moving all that to xfs_defer_finish_one.
--- libxfs/xfs_defer.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-)
libxfs/xfs_defer.c +27 -1
@@ -186,6 +186,32 @@ static const struct xfs_defer_op_type *defer_op_types[] = { [XFS_DEFER_OPS_TYPE_ATTR] = &xfs_attr_defer_type, }; +/* Create a log intent done item for a log intent item. */ +static inline void +xfs_defer_create_done ( +struct xfs_trans *tp, +struct xfs_defer_pending *dfp) +{ +const struct xfs_defer_op_type *ops = defer_op_types[dfp->dfp_type]; +struct xfs_log_item *lip; + +/* +* Mark the transaction dirty, even on error. This ensures the +* transaction is aborted, which: +* +* 1.) releases the log intent item and frees the log done item +* 2.) shuts down the filesystem +*/ + tp->t_flags |=XFS_TRANS_DIRTY ; + lip = ops->create_done (tp, dfp->dfp_intent, dfp->dfp_count); +if (!lip) +return ; + + tp->t_flags |=XFS_TRANS_HAS_INTENT_DONE ; +set_bit (XFS_LI_DIRTY , &lip->li_flags); + dfp->dfp_done = lip; +} +/* * Ensure there's a log intent item associated with this deferred work item if * the operation must be restarted on crash. Returns 1 if there's a log item;
@@ -491,7 +517,7 @@ xfs_defer_finish_one(trace_xfs_defer_pending_finish (tp->t_mountp, dfp); - dfp->dfp_done = ops->create_done (tp, dfp->dfp_intent, dfp->dfp_count); +xfs_defer_create_done (tp, dfp);list_for_each_safe (li, n, &dfp->dfp_work) {list_del (li); dfp->dfp_count--;
PATCH 6/67 xfs: hoist ->create_intent boilerplate to its callsite
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: f3fd7f6fce1cc9b8eb59705b27f823330207b7c9
Hoist the dirty flag setting code out of each ->create_intent implementation up to the callsite to reduce boilerplate further.
--- libxfs/xfs_defer.c | 2 ++ 1 file changed, 2 insertions(+)
libxfs/xfs_defer.c +2 -0
@@ -235,6 +235,8 @@ xfs_defer_create_intent(if (IS_ERR (lip))return PTR_ERR (lip); + tp->t_flags |=XFS_TRANS_DIRTY ; +set_bit (XFS_LI_DIRTY , &lip->li_flags); dfp->dfp_intent = lip;return 1 ; }
PATCH 7/67 xfs: use xfs_defer_create_done for the relogging operation
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: bd3a88f6b71c7509566b44b7021581191cc11ae3
Now that we have a helper to handle creating a log intent done item and updating all the necessary state flags, use it to reduce boilerplate in the ->iop_relog implementations.
--- include/xfs_trans.h | 2 +- libxfs/xfs_defer.c | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-)
include/xfs_trans.h +1 -1
@@ -158,7 +158,7 @@ libxfs_trans_read_buf( }#define xfs_log_item_in_current_chkpt (lip) (false) -#define xfs_trans_item_relog (lip, tp) (NULL) +#define xfs_trans_item_relog (lip, dontcare, tp) (NULL)/* Contorted mess to make gcc shut up about unused vars. */ #define xlog_grant_push_threshold (log, need) \
libxfs/xfs_defer.c +5 -1
@@ -495,7 +495,11 @@ xfs_defer_relog(trace_xfs_defer_relog_intent ((*tpp)->t_mountp, dfp);XFS_STATS_INC ((*tpp)->t_mountp, defer_relog); - dfp->dfp_intent =xfs_trans_item_relog (dfp->dfp_intent, *tpp); + +xfs_defer_create_done (*tpp, dfp); + dfp->dfp_intent =xfs_trans_item_relog (dfp->dfp_intent, + dfp->dfp_done, *tpp); + dfp->dfp_done =NULL ; }if ((*tpp)->t_flags &XFS_TRANS_DIRTY )
PATCH 8/67 xfs: clean out XFS_LI_DIRTY setting boilerplate from ->iop_relog
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 3e0958be2156d90ef908a1a547b4e27a3ec38da9
Hoist this dirty flag setting to the ->iop_relog callsite to reduce boilerplate.
--- libxfs/xfs_defer.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
libxfs/xfs_defer.c +7 -2
@@ -469,6 +469,8 @@ xfs_defer_relog(ASSERT ((*tpp)->t_flags &XFS_TRANS_PERM_LOG_RES );list_for_each_entry (dfp, dfops, dfp_list) { +struct xfs_log_item *lip; +/* * If the log intent item for this deferred op is not a part of * the current log checkpoint, relog the intent item to keep
@@ -497,9 +499,12 @@ xfs_defer_relog(XFS_STATS_INC ((*tpp)->t_mountp, defer_relog);xfs_defer_create_done (*tpp, dfp); - dfp->dfp_intent =xfs_trans_item_relog (dfp->dfp_intent, - dfp->dfp_done, *tpp); + lip =xfs_trans_item_relog (dfp->dfp_intent, dfp->dfp_done, + *tpp); +if (lip) +set_bit (XFS_LI_DIRTY , &lip->li_flags); dfp->dfp_done =NULL ; + dfp->dfp_intent = lip; }if ((*tpp)->t_flags &XFS_TRANS_DIRTY )
PATCH 9/67 xfs: hoist xfs_trans_add_item calls to defer ops functions
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: b28852a5bd08654634e4e32eb072fba14c5fae26
Remove even more repeated boilerplate.
--- libxfs/xfs_defer.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
libxfs/xfs_defer.c +5 -1
@@ -208,6 +208,7 @@ xfs_defer_create_done(return ; tp->t_flags |=XFS_TRANS_HAS_INTENT_DONE ; +xfs_trans_add_item (tp, lip);set_bit (XFS_LI_DIRTY , &lip->li_flags); dfp->dfp_done = lip; }
@@ -236,6 +237,7 @@ xfs_defer_create_intent(return PTR_ERR (lip); tp->t_flags |=XFS_TRANS_DIRTY ; +xfs_trans_add_item (tp, lip);set_bit (XFS_LI_DIRTY , &lip->li_flags); dfp->dfp_intent = lip;return 1 ;
@@ -501,8 +503,10 @@ xfs_defer_relog(xfs_defer_create_done (*tpp, dfp); lip =xfs_trans_item_relog (dfp->dfp_intent, dfp->dfp_done, *tpp); -if (lip) +if (lip) { +xfs_trans_add_item (*tpp, lip);set_bit (XFS_LI_DIRTY , &lip->li_flags); + } dfp->dfp_done =NULL ; dfp->dfp_intent = lip; }
PATCH 10/67 xfs: move ->iop_relog to struct xfs_defer_op_type
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: a49c708f9a445457f6a5905732081871234f61c6
The only log items that need relogging are the ones created for deferred work operations, and the only part of the code base that relogs log items is the deferred work machinery. Move the function pointers.
--- include/xfs_trans.h | 1 - libxfs/xfs_defer.c | 31 ++++++++++++++++++++----------- libxfs/xfs_defer.h | 3 +++ 3 files changed, 23 insertions(+), 12 deletions(-)
include/xfs_trans.h +0 -1
@@ -158,7 +158,6 @@ libxfs_trans_read_buf( }#define xfs_log_item_in_current_chkpt (lip) (false) -#define xfs_trans_item_relog (lip, dontcare, tp) (NULL)/* Contorted mess to make gcc shut up about unused vars. */ #define xlog_grant_push_threshold (log, need) \
libxfs/xfs_defer.c +20 -11
@@ -453,6 +453,25 @@ xfs_defer_cancel_list(xfs_defer_pending_cancel_work (mp, dfp); } +static inline void +xfs_defer_relog_intent ( +struct xfs_trans *tp, +struct xfs_defer_pending *dfp) +{ +struct xfs_log_item *lip; +const struct xfs_defer_op_type *ops = defer_op_types[dfp->dfp_type]; + +xfs_defer_create_done (tp, dfp); + + lip = ops->relog_intent (tp, dfp->dfp_intent, dfp->dfp_done); +if (lip) { +xfs_trans_add_item (tp, lip); +set_bit (XFS_LI_DIRTY , &lip->li_flags); + } + dfp->dfp_done =NULL ; + dfp->dfp_intent = lip; +} +/* * Prevent a log intent item from pinning the tail of the log by logging a * done item to release the intent item; and then log a new intent item.
@@ -471,8 +490,6 @@ xfs_defer_relog(ASSERT ((*tpp)->t_flags &XFS_TRANS_PERM_LOG_RES );list_for_each_entry (dfp, dfops, dfp_list) { -struct xfs_log_item *lip; -/* * If the log intent item for this deferred op is not a part of * the current log checkpoint, relog the intent item to keep
@@ -500,15 +517,7 @@ xfs_defer_relog(trace_xfs_defer_relog_intent ((*tpp)->t_mountp, dfp);XFS_STATS_INC ((*tpp)->t_mountp, defer_relog); -xfs_defer_create_done (*tpp, dfp); - lip =xfs_trans_item_relog (dfp->dfp_intent, dfp->dfp_done, - *tpp); -if (lip) { -xfs_trans_add_item (*tpp, lip); -set_bit (XFS_LI_DIRTY , &lip->li_flags); - } - dfp->dfp_done =NULL ; - dfp->dfp_intent = lip; +xfs_defer_relog_intent (*tpp, dfp); }if ((*tpp)->t_flags &XFS_TRANS_DIRTY )
libxfs/xfs_defer.h +3 -0
@@ -59,6 +59,9 @@ struct xfs_defer_op_type {void (*cancel_item)(struct list_head *item);int (*recover_work)(struct xfs_defer_pending *dfp,struct list_head *capture_list); +struct xfs_log_item *(*relog_intent)(struct xfs_trans *tp, +struct xfs_log_item *intent, +struct xfs_log_item *done_item);unsigned int max_items; };
PATCH 11/67 xfs: make rextslog computation consistent with mkfs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: a6a38f309afc4a7ede01242b603f36c433997780
There's a weird discrepancy in xfsprogs dating back to the creation of the Linux port -- if there are zero rt extents, mkfs will set sb_rextents and sb_rextslog both to zero:
sbp->sb_rextslog =
(uint8_t)(rtextents ?libxfs_highbit32((unsigned int)rtextents) : 0);
However, that's not the check that xfs_repair uses for nonzero rtblocks:
if (sb->sb_rextslog !=libxfs_highbit32((unsigned int)sb->sb_rextents))
The difference here is that xfs_highbit32 returns -1 if its argument is zero. Unfortunately, this means that in the weird corner case of a realtime volume shorter than 1 rt extent, xfs_repair will immediately flag a freshly formatted filesystem as corrupt. Because mkfs has been writing ondisk artifacts like this for decades, we have to accept that as "correct". TBH, zero rextslog for zero rtextents makes more sense to me anyway.
Regrettably, the superblock verifier checks created in commit copied xfs_repair even though mkfs has been writing out such filesystems for ages. Fix the superblock verifier to accept what mkfs spits out; the userspace version of this patch will have to fix xfs_repair as well.
Note that the new helper leaves the zeroday bug where the upper 32 bits of sb_rextents is ripped off and fed to highbit32. This leads to a seriously undersized rt summary file, which immediately breaks mkfs:
$ hugedisk.sh foo /dev/sdc $(( 0x100000080 * 4096))B $ /sbin/mkfs.xfs -f /dev/sda -m rmapbt=0,reflink=0 -r rtdev=/dev/mapper/foo meta-data=/dev/sda isize=512 agcount=4, agsize=1298176 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 bigtime=1 inobtcount=1 nrext64=1 data = bsize=4096 blocks=5192704, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=16384, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =/dev/mapper/foo extsz=4096 blocks=4294967424, rtextents=4294967424 Discarding blocks...Done. mkfs.xfs: Error initializing the realtime space [117 - Structure needs cleaning]
The next patch will drop support for rt volumes with fewer than 1 or more than 2^32-1 rt extents, since they've clearly been broken forever.
--- libxfs/libxfs_api_defs.h | 1 + libxfs/xfs_rtbitmap.c | 12 ++++++++++++ libxfs/xfs_rtbitmap.h | 3 +++ libxfs/xfs_sb.c | 3 ++- mkfs/xfs_mkfs.c | 3 +-- repair/sb.c | 3 +-- 6 files changed, 20 insertions(+), 5 deletions(-)
libxfs/libxfs_api_defs.h +1 -0
@@ -64,6 +64,7 @@#define xfs_bunmapi libxfs_bunmapi#define xfs_bwrite libxfs_bwrite#define xfs_calc_dquots_per_chunk libxfs_calc_dquots_per_chunk +#define xfs_compute_rextslog libxfs_compute_rextslog#define xfs_da3_node_hdr_from_disk libxfs_da3_node_hdr_from_disk#define xfs_da_get_buf libxfs_da_get_buf#define xfs_da_hashname libxfs_da_hashname
libxfs/xfs_rtbitmap.c +12 -0
@@ -1128,6 +1128,18 @@ xfs_rtbitmap_blockcount(return howmany_64 (rtextents,NBBY * mp->m_sb.sb_blocksize); } +/* +* Compute the maximum level number of the realtime summary file, as defined by +* mkfs. The use of highbit32 on a 64-bit quantity is a historic artifact that +* prohibits correct use of rt volumes with more than 2^32 extents. +*/ +uint8_t +xfs_compute_rextslog ( +xfs_rtbxlen_t rtextents) +{ +return rtextents ?xfs_highbit32 (rtextents) :0 ; +} +/* * Compute the number of rtbitmap words needed to populate every block of a * bitmap that is large enough to track the given number of rt extents.
libxfs/xfs_rtbitmap.h +3 -0
@@ -351,6 +351,8 @@ xfs_rtfree_extent(int xfs_rtfree_blocks (struct xfs_trans *tp,xfs_fsblock_t rtbno,xfs_filblks_t rtlen); +uint8_t xfs_compute_rextslog (xfs_rtbxlen_t rtextents); +xfs_filblks_t xfs_rtbitmap_blockcount (struct xfs_mount *mp,xfs_rtbxlen_t rtextents); unsigned long long xfs_rtbitmap_wordcount(struct xfs_mount *mp,
@@ -369,6 +371,7 @@ unsigned long long xfs_rtsummary_wordcount(struct xfs_mount *mp,# define xfs_rtsummary_read_buf (a,b) (-ENOSYS)# define xfs_rtbuf_cache_relse (a) (0)# define xfs_rtalloc_extent_is_free (m,t,s,l,i) (-ENOSYS) +# define xfs_compute_rextslog (rtx) (0)static inline xfs_filblks_t xfs_rtbitmap_blockcount (struct xfs_mount *mp,xfs_rtbxlen_t rtextents) {
libxfs/xfs_sb.c +2 -1
@@ -23,6 +23,7 @@#include "xfs_da_format.h" #include "xfs_health.h" #include "xfs_ag.h" +#include "xfs_rtbitmap.h" /* * Physical superblock buffer manipulations. Shared with libxfs in userspace.
@@ -507,7 +508,7 @@ xfs_validate_sb_common( NBBY * sbp->sb_blocksize);if (sbp->sb_rextents != rexts || - sbp->sb_rextslog !=xfs_highbit32 (sbp->sb_rextents) || + sbp->sb_rextslog !=xfs_compute_rextslog (rexts) || sbp->sb_rbmblocks != rbmblocks) {xfs_notice (mp,"realtime geometry sanity check failed" );
mkfs/xfs_mkfs.c +1 -2
@@ -3699,8 +3699,7 @@ finish_superblock_setup( sbp->sb_agcount = (xfs_agnumber_t )cfg->agcount; sbp->sb_rbmblocks = cfg->rtbmblocks; sbp->sb_logblocks = (xfs_extlen_t )cfg->logblocks; - sbp->sb_rextslog = (uint8_t )(cfg->rtextents ? -libxfs_highbit32 ((unsigned int )cfg->rtextents) :0 ); + sbp->sb_rextslog =libxfs_compute_rextslog (cfg->rtextents); sbp->sb_inprogress =1 ;/* mkfs is in progress */ sbp->sb_imax_pct = cfg->imaxpct; sbp->sb_icount =0 ;
repair/sb.c +1 -2
@@ -475,8 +475,7 @@ verify_sb(char *sb_buf, xfs_sb_t *sb, int is_primary_sb)if (sb->sb_rblocks / sb->sb_rextsize != sb->sb_rextents)return (XR_BAD_RT_GEO_DATA ); -if (sb->sb_rextslog != -libxfs_highbit32 ((unsigned int )sb->sb_rextents)) +if (sb->sb_rextslog !=libxfs_compute_rextslog (sb->sb_rextents))return (XR_BAD_RT_GEO_DATA ); if (sb->sb_rbmblocks != (xfs_extlen_t) howmany(sb->sb_rextents,
PATCH 12/67 xfs: fix 32-bit truncation in xfs_compute_rextslog
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: cf8f0e6c1429be7652869059ea44696b72d5b726
It's quite reasonable that some customer somewhere will want to configure a realtime volume with more than 2^32 extents. If they try to do this, the highbit32() call will truncate the upper bits of the xfs_rtbxlen_t and produce the wrong value for rextslog. This in turn causes the rsumlevels to be wrong, which results in a realtime summary file that is the wrong length. Fix that.
--- libxfs/xfs_rtbitmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
libxfs/xfs_rtbitmap.c +5 -3
@@ -1130,14 +1130,16 @@ xfs_rtbitmap_blockcount(/* * Compute the maximum level number of the realtime summary file, as defined by -* mkfs. The use of highbit32 on a 64-bit quantity is a historic artifact that -* prohibits correct use of rt volumes with more than 2^32 extents. +* mkfs. The historic use of highbit32 on a 64-bit quantity prohibited correct +* use of rt volumes with more than 2^32 extents. */ uint8_t xfs_compute_rextslog (xfs_rtbxlen_t rtextents) { -return rtextents ?xfs_highbit32 (rtextents) :0 ; +if (!rtextents) +return 0 ; +return xfs_highbit64 (rtextents); }/*
PATCH 13/67 xfs: don't allow overly small or large realtime volumes
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: e14293803f4e84eb23a417b462b56251033b5a66
Don't allow realtime volumes that are less than one rt extent long. This has been broken across 4 LTS kernels with nobody noticing, so let's just disable it.
--- libxfs/xfs_rtbitmap.h | 13 +++++++++++++ libxfs/xfs_sb.c | 3 ++- mkfs/xfs_mkfs.c | 5 +++++ repair/sb.c | 3 +++ 4 files changed, 23 insertions(+), 1 deletion(-)
libxfs/xfs_rtbitmap.h +13 -0
@@ -353,6 +353,18 @@ int xfs_rtfree_blocks(struct xfs_trans *tp, xfs_fsblock_t rtbno,uint8_t xfs_compute_rextslog (xfs_rtbxlen_t rtextents); +/* Do we support an rt volume having this number of rtextents? */ +static inline bool +xfs_validate_rtextents ( +xfs_rtbxlen_t rtextents) +{ +/* No runt rt volumes */ +if (rtextents ==0 ) +return false; + +return true; +} +xfs_filblks_t xfs_rtbitmap_blockcount (struct xfs_mount *mp,xfs_rtbxlen_t rtextents); unsigned long long xfs_rtbitmap_wordcount(struct xfs_mount *mp,
@@ -372,6 +384,7 @@ unsigned long long xfs_rtsummary_wordcount(struct xfs_mount *mp,# define xfs_rtbuf_cache_relse (a) (0)# define xfs_rtalloc_extent_is_free (m,t,s,l,i) (-ENOSYS)# define xfs_compute_rextslog (rtx) (0) +# define xfs_validate_rtextents (rtx) (false)static inline xfs_filblks_t xfs_rtbitmap_blockcount (struct xfs_mount *mp,xfs_rtbxlen_t rtextents) {
libxfs/xfs_sb.c +2 -1
@@ -507,7 +507,8 @@ xfs_validate_sb_common( rbmblocks =howmany_64 (sbp->sb_rextents,NBBY * sbp->sb_blocksize); -if (sbp->sb_rextents != rexts || +if (!xfs_validate_rtextents (rexts) || + sbp->sb_rextents != rexts || sbp->sb_rextslog !=xfs_compute_rextslog (rexts) || sbp->sb_rbmblocks != rbmblocks) { xfs_notice(mp,
mkfs/xfs_mkfs.c +5 -0
@@ -3025,6 +3025,11 @@ reported by the device (%u).\n"), } cfg->rtextents = cfg->rtblocks / cfg->rtextblocks; +if (cfg->rtextents ==0 ) { +fprintf (stderr, +_ ("cannot have an rt subvolume with zero extents\n" )); +usage (); + } cfg->rtbmblocks = (xfs_extlen_t )howmany (cfg->rtextents,NBBY * cfg->blocksize); }
repair/sb.c +3 -0
@@ -475,6 +475,9 @@ verify_sb(char *sb_buf, xfs_sb_t *sb, int is_primary_sb)if (sb->sb_rblocks / sb->sb_rextsize != sb->sb_rextents)return (XR_BAD_RT_GEO_DATA ); +if (sb->sb_rextents ==0 ) +return XR_BAD_RT_GEO_DATA ; +if (sb->sb_rextslog !=libxfs_compute_rextslog (sb->sb_rextents))return (XR_BAD_RT_GEO_DATA );
PATCH 14/67 xfs: elide ->create_done calls for unlogged deferred work
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 9c07bca793b4ff9f0b7871e2a928a1b28b8fa4e3
Extended attribute updates use the deferred work machinery to manage state across a chain of smaller transactions. All previous deferred work users have employed log intent items and log done items to manage restarting of interrupted operations, which means that ->create_intent sets dfp_intent to a log intent item and ->create_done uses that item to create a log intent done item.
However, xattrs have used the INCOMPLETE flag to deal with the lack of recovery support for an interrupted transaction chain. Log items are optional if the xattr update caller didn't set XFS_DA_OP_LOGGED to require a restartable sequence.
In other words, ->create_intent can return NULL to say that there's no log intent item. If that's the case, no log intent done item should be created. Clean up xfs_defer_create_done not to do this, so that the ->create_done functions don't have to check for non-null dfp_intent themselves.
--- libxfs/xfs_defer.c | 4 ++++ 1 file changed, 4 insertions(+)
libxfs/xfs_defer.c +4 -0
@@ -195,6 +195,10 @@ xfs_defer_create_done(const struct xfs_defer_op_type *ops = defer_op_types[dfp->dfp_type];struct xfs_log_item *lip; +/* If there is no log intent item, there can be no log done item. */ +if (!dfp->dfp_intent) +return ; +/* * Mark the transaction dirty, even on error. This ensures the * transaction is aborted, which:
PATCH 15/67 xfs: don't append work items to logged xfs_defer_pending objects
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 6b126139401a2284402d7c38fe3168d5a26da41d
When someone tries to add a deferred work item to xfs_defer_add, it will try to attach the work item to the most recently added xfs_defer_pending object attached to the transaction. However, it doesn't check if the pending object has a log intent item attached to it. This is incorrect behavior because we cannot add more work to an object that has already been committed to the ondisk log.
Therefore, change the behavior not to append to pending items with a non null dfp_intent. In practice this has not been an issue because the only way xfs_defer_add gets called after log intent items have been the @dop_pending isolation in xfs_defer_finish_noroll protects the pending items that have already been logged.
However, the next patch will add the ability to pause a deferred extent free object during online btree rebuilding, and any new extfree work items need to have their own pending event.
While we're at it, hoist the predicate to its own static inline function for readability.
--- libxfs/xfs_defer.c | 61 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 13 deletions(-)
libxfs/xfs_defer.c +48 -13
@@ -689,6 +689,51 @@ xfs_defer_cancel(xfs_defer_cancel_list (mp, &tp->t_dfops); } +/* +* Return the last pending work item attached to this transaction if it matches +* the deferred op type. +*/ +static inline struct xfs_defer_pending * +xfs_defer_find_last ( +struct xfs_trans *tp, +enum xfs_defer_ops_type type, +const struct xfs_defer_op_type *ops) +{ +struct xfs_defer_pending *dfp =NULL ; + +/* No dfops at all? */ +if (list_empty (&tp->t_dfops)) +return NULL ; + + dfp = list_last_entry(&tp->t_dfops, struct xfs_defer_pending, + dfp_list); + +/* Wrong type? */ +if (dfp->dfp_type != type) +return NULL ; +return dfp; +} + +/* +* Decide if we can add a deferred work item to the last dfops item attached +* to the transaction. +*/ +static inline bool +xfs_defer_can_append ( +struct xfs_defer_pending *dfp, +const struct xfs_defer_op_type *ops) +{ +/* Already logged? */ +if (dfp->dfp_intent) +return false; + +/* Already full? */ +if (ops->max_items && dfp->dfp_count >= ops->max_items) +return false; + +return true; +} +/* Add an item for later deferred processing. */ void xfs_defer_add(
@@ -702,19 +747,9 @@ xfs_defer_add(ASSERT (tp->t_flags &XFS_TRANS_PERM_LOG_RES );BUILD_BUG_ON (ARRAY_SIZE (defer_op_types) !=XFS_DEFER_OPS_TYPE_MAX ); -/* -* Add the item to a pending item at the end of the intake list. -* If the last pending item has the same type, reuse it. Else, -* create a new pending item at the end of the intake list. -*/ -if (!list_empty (&tp->t_dfops)) { - dfp =list_last_entry (&tp->t_dfops, - struct xfs_defer_pending, dfp_list); -if (dfp->dfp_type != type || - (ops->max_items && dfp->dfp_count >= ops->max_items)) - dfp =NULL ; - } -if (!dfp) { + dfp =xfs_defer_find_last (tp, type, ops); +if (!dfp || !xfs_defer_can_append (dfp, ops)) { +/* Create a new pending item at the end of the intake list. */ dfp =kmem_cache_zalloc (xfs_defer_pending_cache,GFP_NOFS | __GFP_NOFAIL); dfp->dfp_type = type;
PATCH 16/67 xfs: allow pausing of pending deferred work items
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 4dffb2cbb4839fd6f9bbac0b3fd06cc9015cbb9b
Traditionally, all pending deferred work attached to a transaction is finished when one of the xfs_defer_finish* functions is called. However, online repair wants to be able to allocate space for a new data structure, format a new metadata structure into the allocated space, and
As a hedge against system crashes during repairs, we also want to log some EFI items for the allocated space speculatively, and cancel them if we elect to commit the new data structure.
Therefore, introduce the idea of pausing a pending deferred work item. Log intent items are still created for paused items and relogged as necessary. However, paused items are pushed onto a side list before we start calling ->finish_item, and the whole list is reattach to the transaction afterwards. New work items are never attached to paused pending items.
Modify xfs_defer_cancel to clean up pending deferred work items holding a log intent item but not a log intent done item, since that is now possible.
--- include/list.h | 14 +++++++ include/xfs_trace.h | 3 ++ libxfs/xfs_defer.c | 97 +++++++++++++++++++++++++++++++++++++++++++++------ libxfs/xfs_defer.h | 17 ++++++++- 4 files changed, 117 insertions(+), 14 deletions(-)
include/list.h +14 -0
@@ -152,6 +152,20 @@ static inline void list_splice_init(struct list_head *list,#define list_first_entry (ptr, type, member) \ list_entry((ptr)->next, type, member) +/** +* list_first_entry_or_null - get the first element from a list +* @ptr: the list head to take the element from. +* @type: the type of the struct this is embedded in. +* @member: the name of the list_head within the struct. +* +* Note that if the list is empty, it returns NULL. +*/ +#define list_first_entry_or_null (ptr, type, member) ({ \ + struct list_head *head__ = (ptr); \ + struct list_head *pos__ = head__->next; \ + pos__ != head__ ? list_entry(pos__, type, member) : NULL; \ +}) +#define container_of (ptr, type, member) ({ \ const typeof( ((type *)0)->member ) *__mptr = (ptr); \ (type *)( (char *)__mptr - offsetof(type,member) );})
include/xfs_trace.h +3 -0
@@ -228,6 +228,9 @@#define trace_xfs_defer_finish_done (a,b) ((void) 0)#define trace_xfs_defer_cancel_list (a,b) ((void) 0)#define trace_xfs_defer_create_intent (a,b) ((void) 0) +#define trace_xfs_defer_isolate_paused (...) ((void) 0) +#define trace_xfs_defer_item_pause (...) ((void) 0) +#define trace_xfs_defer_item_unpause (...) ((void) 0)#define trace_xfs_bmap_free_defer (...) ((void) 0)#define trace_xfs_bmap_free_deferred (...) ((void) 0)
libxfs/xfs_defer.c +85 -12
183 lines not on this page · raw message
libxfs/xfs_defer.h +15 -2
@@ -34,11 +34,24 @@ struct xfs_defer_pending {struct xfs_log_item *dfp_intent;/* log intent item */ struct xfs_log_item *dfp_done;/* log done item */ unsigned int dfp_count;/* # extent items */ +unsigned int dfp_flags;enum xfs_defer_ops_type dfp_type; }; -void xfs_defer_add (struct xfs_trans *tp,enum xfs_defer_ops_type type, -struct list_head *h); +/* +* Create a log intent item for this deferred item, but don't actually finish +* the work. Caller must clear this before the final transaction commit. +*/ +#define XFS_DEFER_PAUSED (1U << 0) + +#define XFS_DEFER_PENDING_STRINGS \ + { XFS_DEFER_PAUSED, "paused" } + +void xfs_defer_item_pause (struct xfs_trans *tp,struct xfs_defer_pending *dfp); +void xfs_defer_item_unpause (struct xfs_trans *tp,struct xfs_defer_pending *dfp); + +struct xfs_defer_pending *xfs_defer_add (struct xfs_trans *tp, +enum xfs_defer_ops_type type,struct list_head *h);int xfs_defer_finish_noroll (struct xfs_trans **tp);int xfs_defer_finish (struct xfs_trans **tp);int xfs_defer_finish_one (struct xfs_trans *tp,struct xfs_defer_pending *dfp);
PATCH 17/67 xfs: remove __xfs_free_extent_later
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 4c88fef3af4a51c2cdba6a28237e98da4873e8dc
xfs_free_extent_later is a trivial helper, so remove it to reduce the amount of thinking required to understand the deferred freeing interface. This will make it easier to introduce automatic reaping of speculative allocations in the next patch.
--- libxfs/xfs_ag.c | 2 +- libxfs/xfs_alloc.c | 2 +- libxfs/xfs_alloc.h | 14 +------------- libxfs/xfs_bmap.c | 4 ++-- libxfs/xfs_bmap_btree.c | 2 +- libxfs/xfs_ialloc.c | 5 +++-- libxfs/xfs_ialloc_btree.c | 2 +- libxfs/xfs_refcount.c | 6 +++--- libxfs/xfs_refcount_btree.c | 2 +- 9 files changed, 14 insertions(+), 25 deletions(-)
libxfs/xfs_ag.c +1 -1
@@ -982,7 +982,7 @@ xfs_ag_shrink_space(if (err2 != -ENOSPC ) goto resv_err; - err2 =__xfs_free_extent_later (*tpp, args.fsbno, delta,NULL , + err2 =xfs_free_extent_later (*tpp, args.fsbno, delta,NULL ,XFS_AG_RESV_NONE , true);if (err2) goto resv_err;
libxfs/xfs_alloc.c +1 -1
@@ -2519,7 +2519,7 @@ xfs_defer_agfl_block(* The list is maintained sorted (by block number). */ int -__xfs_free_extent_later( +xfs_free_extent_later( struct xfs_trans *tp, xfs_fsblock_t bno, xfs_filblks_t len,
libxfs/xfs_alloc.h +1 -13
@@ -231,7 +231,7 @@ xfs_buf_to_agfl_bno(return bp->b_addr; } -int __xfs_free_extent_later (struct xfs_trans *tp,xfs_fsblock_t bno, +int xfs_free_extent_later (struct xfs_trans *tp,xfs_fsblock_t bno,xfs_filblks_t len,const struct xfs_owner_info *oinfo,enum xfs_ag_resv_type type,bool skip_discard);
@@ -256,18 +256,6 @@ void xfs_extent_free_get_group(struct xfs_mount *mp,#define XFS_EFI_ATTR_FORK (1U << 1)/* freeing attr fork block */ #define XFS_EFI_BMBT_BLOCK (1U << 2)/* freeing bmap btree block */ -static inline int -xfs_free_extent_later ( -struct xfs_trans *tp, -xfs_fsblock_t bno, -xfs_filblks_t len, -const struct xfs_owner_info *oinfo, -enum xfs_ag_resv_type type) -{ -return __xfs_free_extent_later (tp, bno, len, oinfo, type, false); -} - -extern struct kmem_cache *xfs_extfree_item_cache;int __init xfs_extfree_intent_init_cache (void );
libxfs/xfs_bmap.c +2 -2
@@ -569,7 +569,7 @@ xfs_bmap_btree_to_extents(xfs_rmap_ino_bmbt_owner (&oinfo, ip->i_ino, whichfork); error =xfs_free_extent_later (cur->bc_tp, cbno,1 , &oinfo, -XFS_AG_RESV_NONE ); +XFS_AG_RESV_NONE , false);if (error)return error;
@@ -5212,7 +5212,7 @@ xfs_bmap_del_extent_real(if (xfs_is_reflink_inode (ip) && whichfork ==XFS_DATA_FORK ) {xfs_refcount_decrease_extent (tp, del); }else { - error = __xfs_free_extent_later(tp, del->br_startblock, + error = xfs_free_extent_later(tp, del->br_startblock, del->br_blockcount, NULL, XFS_AG_RESV_NONE, ((bflags & XFS_BMAPI_NODISCARD) ||
libxfs/xfs_bmap_btree.c +1 -1
@@ -270,7 +270,7 @@ xfs_bmbt_free_block(xfs_rmap_ino_bmbt_owner (&oinfo, ip->i_ino, cur->bc_ino.whichfork); error =xfs_free_extent_later (cur->bc_tp, fsbno,1 , &oinfo, -XFS_AG_RESV_NONE ); +XFS_AG_RESV_NONE , false);if (error)return error;
libxfs/xfs_ialloc.c +3 -2
@@ -1849,7 +1849,7 @@ xfs_difree_inode_chunk(return xfs_free_extent_later (tp,XFS_AGB_TO_FSB (mp, agno, sagbno),M_IGEO (mp)->ialloc_blks, &XFS_RMAP_OINFO_INODES , -XFS_AG_RESV_NONE ); +XFS_AG_RESV_NONE , false); }/* holemask is only 16-bits (fits in an unsigned long) */
@@ -1895,7 +1895,8 @@ xfs_difree_inode_chunk(ASSERT (contigblk % mp->m_sb.sb_spino_align ==0 ); error =xfs_free_extent_later (tp,XFS_AGB_TO_FSB (mp, agno, agbno), contigblk, - &XFS_RMAP_OINFO_INODES ,XFS_AG_RESV_NONE ); + &XFS_RMAP_OINFO_INODES ,XFS_AG_RESV_NONE , + false);if (error)return error;
libxfs/xfs_ialloc_btree.c +1 -1
@@ -160,7 +160,7 @@ __xfs_inobt_free_block(xfs_inobt_mod_blockcount (cur,-1 ); fsbno =XFS_DADDR_TO_FSB (cur->bc_mp,xfs_buf_daddr (bp));return xfs_free_extent_later (cur->bc_tp, fsbno,1 , - &XFS_RMAP_OINFO_INOBT , resv); + &XFS_RMAP_OINFO_INOBT , resv, false); }STATIC int
libxfs/xfs_refcount.c +3 -3
@@ -1152,7 +1152,7 @@ xfs_refcount_adjust_extents( tmp.rc_startblock); error =xfs_free_extent_later (cur->bc_tp, fsbno, tmp.rc_blockcount,NULL , -XFS_AG_RESV_NONE ); +XFS_AG_RESV_NONE , false);if (error) goto out_error; }
@@ -1214,7 +1214,7 @@ xfs_refcount_adjust_extents( ext.rc_startblock); error =xfs_free_extent_later (cur->bc_tp, fsbno, ext.rc_blockcount,NULL , -XFS_AG_RESV_NONE ); +XFS_AG_RESV_NONE , false);if (error) goto out_error; }
@@ -1984,7 +1984,7 @@ xfs_refcount_recover_cow_leftovers(/* Free the block. */ error =xfs_free_extent_later (tp, fsb, rr->rr_rrec.rc_blockcount,NULL , -XFS_AG_RESV_NONE ); +XFS_AG_RESV_NONE , false);if (error) goto out_trans;
libxfs/xfs_refcount_btree.c +1 -1
@@ -111,7 +111,7 @@ xfs_refcountbt_free_block(be32_add_cpu (&agf->agf_refcount_blocks,-1 );xfs_alloc_log_agf (cur->bc_tp, agbp,XFS_AGF_REFCOUNT_BLOCKS );return xfs_free_extent_later (cur->bc_tp, fsbno,1 , - &XFS_RMAP_OINFO_REFC ,XFS_AG_RESV_METADATA ); + &XFS_RMAP_OINFO_REFC ,XFS_AG_RESV_METADATA , false); }STATIC int
PATCH 18/67 xfs: automatic freeing of freshly allocated unwritten space
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: e3042be36c343207b7af249a09f50b4e37e9fda4
As mentioned in the previous commit, online repair wants to allocate space to write out a new metadata structure, and it also wants to hedge against system crashes during repairs by logging (and later cancelling) EFIs to free the space if we crash before committing the new data structure.
Therefore, create a trio of functions to schedule automatic reaping of freshly allocated unwritten space. xfs_alloc_schedule_autoreap creates a paused EFI representing the space we just allocated. Once the allocations are made and the autoreaps scheduled, we can start writing to disk.
If the writes succeed, xfs_alloc_cancel_autoreap marks the EFI work items as stale and unpauses the pending deferred work item. Assuming that's done in the same transaction that commits the new structure into the filesystem, we guarantee that either the new object is fully visible, or that all the space gets reclaimed.
If the writes succeed but only part of an extent was used, repair must call the same _cancel_autoreap function to kill the first EFI and then log a new EFI to free the unused space. The first EFI is already
For full extents that aren't used, xfs_alloc_commit_autoreap will unpause the EFI, which results in the space being freed during the next _defer_finish cycle.
--- libxfs/defer_item.c | 10 +++-- libxfs/xfs_alloc.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++-- libxfs/xfs_alloc.h | 12 ++++++ 3 files changed, 119 insertions(+), 7 deletions(-)
libxfs/defer_item.c +7 -3
@@ -101,7 +101,7 @@ xfs_extent_free_finish_item(struct xfs_owner_info oinfo = { };struct xfs_extent_free_item *xefi;xfs_agblock_t agbno; -int error; +int error =0 ; xefi = container_of(item, struct xfs_extent_free_item, xefi_list);
@@ -112,8 +112,12 @@ xfs_extent_free_finish_item( oinfo.oi_flags |=XFS_OWNER_INFO_BMBT_BLOCK ; agbno =XFS_FSB_TO_AGBNO (tp->t_mountp, xefi->xefi_startblock); - error =xfs_free_extent (tp, xefi->xefi_pag, agbno, - xefi->xefi_blockcount, &oinfo,XFS_AG_RESV_NONE ); + +if (!(xefi->xefi_flags &XFS_EFI_CANCELLED )) { + error =xfs_free_extent (tp, xefi->xefi_pag, agbno, + xefi->xefi_blockcount, &oinfo, +XFS_AG_RESV_NONE ); + }/* * Don't free the XEFI if we need a new transaction to complete
libxfs/xfs_alloc.c +100 -4
124 lines not on this page · raw message
libxfs/xfs_alloc.h +12 -0
@@ -255,6 +255,18 @@ void xfs_extent_free_get_group(struct xfs_mount *mp,#define XFS_EFI_SKIP_DISCARD (1U << 0)/* don't issue discard */ #define XFS_EFI_ATTR_FORK (1U << 1)/* freeing attr fork block */ #define XFS_EFI_BMBT_BLOCK (1U << 2)/* freeing bmap btree block */ +#define XFS_EFI_CANCELLED (1U << 3)/* dont actually free the space */ + +struct xfs_alloc_autoreap { +struct xfs_defer_pending *dfp; +}; + +int xfs_alloc_schedule_autoreap (const struct xfs_alloc_arg *args, +bool skip_discard,struct xfs_alloc_autoreap *aarp); +void xfs_alloc_cancel_autoreap (struct xfs_trans *tp, +struct xfs_alloc_autoreap *aarp); +void xfs_alloc_commit_autoreap (struct xfs_trans *tp, +struct xfs_alloc_autoreap *aarp);extern struct kmem_cache *xfs_extfree_item_cache;
PATCH 19/67 xfs: remove unused fields from struct xbtree_ifakeroot
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 4c8ecd1cfdd01fb727121035014d9f654a30bdf2
Remove these unused fields since nobody uses them. They should have been removed years ago in a different cleanup series from Christoph Hellwig.
--- libxfs/xfs_btree_staging.h | 6 ------ 1 file changed, 6 deletions(-)
libxfs/xfs_btree_staging.h +0 -6
@@ -37,12 +37,6 @@ struct xbtree_ifakeroot {/* Number of bytes available for this fork in the inode. */ unsigned int if_fork_size; - -/* Fork format. */ -unsigned int if_format; - -/* Number of records. */ -unsigned int if_extents; };/* Cursor interactions with fake roots for inode-rooted btrees. */
PATCH 20/67 xfs: force small EFIs for reaping btree extents
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 3f3cec031099c37513727efc978a12b6346e326d
Introduce the concept of a defer ops barrier to separate consecutively queued pending work items of the same type. With a barrier in place, the two work items will be tracked separately, and receive separate log intent items. The goal here is to prevent reaping of old metadata blocks from creating unnecessarily huge EFIs that could then run the risk of overflowing the scrub transaction.
--- libxfs/xfs_defer.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++----- libxfs/xfs_defer.h | 3 + 2 files changed, 99 insertions(+), 11 deletions(-)
libxfs/xfs_defer.c +96 -11
@@ -176,6 +176,58 @@ static struct kmem_cache *xfs_defer_pending_cache;* Note that the continuation requested between t2 and t3 is likely to * reoccur. */ +STATIC struct xfs_log_item * +xfs_defer_barrier_create_intent ( +struct xfs_trans *tp, +struct list_head *items, +unsigned int count, +bool sort) +{ +return NULL ; +} + +STATIC void +xfs_defer_barrier_abort_intent ( +struct xfs_log_item *intent) +{ +/* empty */ +} + +STATIC struct xfs_log_item * +xfs_defer_barrier_create_done ( +struct xfs_trans *tp, +struct xfs_log_item *intent, +unsigned int count) +{ +return NULL ; +} + +STATIC int +xfs_defer_barrier_finish_item ( +struct xfs_trans *tp, +struct xfs_log_item *done, +struct list_head *item, +struct xfs_btree_cur **state) +{ +ASSERT (0 ); +return -EFSCORRUPTED ; +} + +STATIC void +xfs_defer_barrier_cancel_item ( +struct list_head *item) +{ +ASSERT (0 ); +} + +static const struct xfs_defer_op_type xfs_barrier_defer_type = { + .max_items =1 , + .create_intent = xfs_defer_barrier_create_intent, + .abort_intent = xfs_defer_barrier_abort_intent, + .create_done = xfs_defer_barrier_create_done, + .finish_item = xfs_defer_barrier_finish_item, + .cancel_item = xfs_defer_barrier_cancel_item, +};static const struct xfs_defer_op_type *defer_op_types[] = { [XFS_DEFER_OPS_TYPE_BMAP ] = &xfs_bmap_update_defer_type,
@@ -184,6 +236,7 @@ static const struct xfs_defer_op_type *defer_op_types[] = { [XFS_DEFER_OPS_TYPE_FREE] = &xfs_extent_free_defer_type, [XFS_DEFER_OPS_TYPE_AGFL_FREE] = &xfs_agfl_free_defer_type, [XFS_DEFER_OPS_TYPE_ATTR] = &xfs_attr_defer_type, + [XFS_DEFER_OPS_TYPE_BARRIER] = &xfs_barrier_defer_type, };/* Create a log intent done item for a log intent item. */
@@ -773,6 +826,23 @@ xfs_defer_can_append(return true; } +/* Create a new pending item at the end of the transaction list. */ +static inline struct xfs_defer_pending * +xfs_defer_alloc ( +struct xfs_trans *tp, +enum xfs_defer_ops_type type) +{ +struct xfs_defer_pending *dfp; + + dfp =kmem_cache_zalloc (xfs_defer_pending_cache, +GFP_NOFS | __GFP_NOFAIL); + dfp->dfp_type = type; +INIT_LIST_HEAD (&dfp->dfp_work); +list_add_tail (&dfp->dfp_list, &tp->t_dfops); + +return dfp; +} +/* Add an item for later deferred processing. */ struct xfs_defer_pending * xfs_defer_add(
@@ -787,23 +857,38 @@ xfs_defer_add(BUILD_BUG_ON (ARRAY_SIZE (defer_op_types) !=XFS_DEFER_OPS_TYPE_MAX ); dfp =xfs_defer_find_last (tp, type, ops); -if (!dfp || !xfs_defer_can_append (dfp, ops)) { -/* Create a new pending item at the end of the intake list. */ - dfp =kmem_cache_zalloc (xfs_defer_pending_cache, -GFP_NOFS | __GFP_NOFAIL); - dfp->dfp_type = type; - dfp->dfp_intent =NULL ; - dfp->dfp_done =NULL ; - dfp->dfp_count =0 ; -INIT_LIST_HEAD (&dfp->dfp_work); -list_add_tail (&dfp->dfp_list, &tp->t_dfops); - } +if (!dfp || !xfs_defer_can_append (dfp, ops)) + dfp =xfs_defer_alloc (tp, type);xfs_defer_add_item (dfp, li);trace_xfs_defer_add_item (tp->t_mountp, dfp, li);return dfp; } +/* +* Add a defer ops barrier to force two otherwise adjacent deferred work items +* to be tracked separately and have separate log items. +*/ +void +xfs_defer_add_barrier ( +struct xfs_trans *tp) +{ +struct xfs_defer_pending *dfp; +const enum xfs_defer_ops_type type =XFS_DEFER_OPS_TYPE_BARRIER ; +const struct xfs_defer_op_type *ops = defer_op_types[type]; + +ASSERT (tp->t_flags &XFS_TRANS_PERM_LOG_RES ); + +/* If the last defer op added was a barrier, we're done. */ + dfp =xfs_defer_find_last (tp, type, ops); +if (dfp) +return ; + +xfs_defer_alloc (tp, type); + +trace_xfs_defer_add_item (tp->t_mountp, dfp,NULL ); +} +/* * Create a pending deferred work item to replay the recovered intent item * and add it to the list.
libxfs/xfs_defer.h +3 -0
@@ -20,6 +20,7 @@ enum xfs_defer_ops_type {XFS_DEFER_OPS_TYPE_FREE ,XFS_DEFER_OPS_TYPE_AGFL_FREE ,XFS_DEFER_OPS_TYPE_ATTR , +XFS_DEFER_OPS_TYPE_BARRIER , XFS_DEFER_OPS_TYPE_MAX, };
@@ -163,4 +164,6 @@ xfs_defer_add_item(int __init xfs_defer_init_item_caches (void );void xfs_defer_destroy_item_caches (void ); +void xfs_defer_add_barrier (struct xfs_trans *tp); +#endif /* __XFS_DEFER_H__ */
PATCH 21/67 xfs: ensure logflagsp is initialized in xfs_bmap_del_extent_real
In reply to Darrick J. WongFrom: Jiachen Zhang <[email protected]>
Source kernel commit: e6af9c98cbf0164a619d95572136bfb54d482dd6
In the case of returning -ENOSPC, ensure logflagsp is initialized by 0. Otherwise the caller __xfs_bunmapi will set uninitialized illegal tmp_logflags value into xfs log, which might cause unpredictable error in the log recovery procedure.
Also, remove the flags variable and set the *logflagsp directly, so that the code should be more robust in the long run.
--- libxfs/xfs_bmap.c | 73 +++++++++++++++++++++++------------------------------ 1 file changed, 31 insertions(+), 42 deletions(-)
libxfs/xfs_bmap.c +31 -42
@@ -5004,7 +5004,6 @@ xfs_bmap_del_extent_real(xfs_fileoff_t del_endoff;/* first offset past del */ int do_fx;/* free extent at end of routine */ int error;/* error return value */ -int flags =0 ;/* inode logging flags */ struct xfs_bmbt_irec got;/* current extent entry */ xfs_fileoff_t got_endoff;/* first offset past got */ int i;/* temp state */
@@ -5017,6 +5016,8 @@ xfs_bmap_del_extent_real(uint32_t state =xfs_bmap_fork_to_state (whichfork);struct xfs_bmbt_irec old; + *logflagsp =0 ; + mp = ip->i_mount;XFS_STATS_INC (mp, xs_del_exlist);
@@ -5029,7 +5030,6 @@ xfs_bmap_del_extent_real(ASSERT (got_endoff >= del_endoff);ASSERT (!isnullstartblock (got.br_startblock)); qfield =0 ; - error =0 ;/* * If it's the case where the directory code is running with no block
@@ -5045,13 +5045,13 @@ xfs_bmap_del_extent_real( del->br_startoff > got.br_startoff && del_endoff < got_endoff)return -ENOSPC ; - flags =XFS_ILOG_CORE ; + *logflagsp =XFS_ILOG_CORE ;if (whichfork ==XFS_DATA_FORK &&XFS_IS_REALTIME_INODE (ip)) {if (!(bflags &XFS_BMAPI_REMAP )) { error =xfs_rtfree_blocks (tp, del->br_startblock, del->br_blockcount);if (error) - goto done; +return error; } do_fx =0 ;
@@ -5066,11 +5066,9 @@ xfs_bmap_del_extent_real(if (cur) { error =xfs_bmbt_lookup_eq (cur, &got, &i);if (error) - goto done; -if (XFS_IS_CORRUPT (mp, i !=1 )) { - error = -EFSCORRUPTED ; - goto done; - } +return error; +if (XFS_IS_CORRUPT (mp, i !=1 )) +return -EFSCORRUPTED ; }if (got.br_startoff == del->br_startoff)
@@ -5087,17 +5085,15 @@ xfs_bmap_del_extent_real(xfs_iext_prev (ifp, icur); ifp->if_nextents--; - flags |=XFS_ILOG_CORE ; + *logflagsp |=XFS_ILOG_CORE ;if (!cur) { - flags |=xfs_ilog_fext (whichfork); + *logflagsp |=xfs_ilog_fext (whichfork);break ; }if ((error =xfs_btree_delete (cur, &i))) - goto done; -if (XFS_IS_CORRUPT (mp, i !=1 )) { - error = -EFSCORRUPTED ; - goto done; - } +return error; +if (XFS_IS_CORRUPT (mp, i !=1 )) +return -EFSCORRUPTED ;break ;case BMAP_LEFT_FILLING :/*
@@ -5108,12 +5104,12 @@ xfs_bmap_del_extent_real( got.br_blockcount -= del->br_blockcount;xfs_iext_update_extent (ip, state, icur, &got);if (!cur) { - flags |=xfs_ilog_fext (whichfork); + *logflagsp |=xfs_ilog_fext (whichfork);break ; } error =xfs_bmbt_update (cur, &got);if (error) - goto done; +return error;break ;case BMAP_RIGHT_FILLING :/*
@@ -5122,12 +5118,12 @@ xfs_bmap_del_extent_real( got.br_blockcount -= del->br_blockcount;xfs_iext_update_extent (ip, state, icur, &got);if (!cur) { - flags |=xfs_ilog_fext (whichfork); + *logflagsp |=xfs_ilog_fext (whichfork);break ; } error =xfs_bmbt_update (cur, &got);if (error) - goto done; +return error;break ;case 0 :/*
@@ -5144,18 +5140,18 @@ xfs_bmap_del_extent_real( new.br_state = got.br_state; new.br_startblock = del_endblock; - flags |=XFS_ILOG_CORE ; + *logflagsp |=XFS_ILOG_CORE ;if (cur) { error =xfs_bmbt_update (cur, &got);if (error) - goto done; +return error; error =xfs_btree_increment (cur,0 , &i);if (error) - goto done; +return error; cur->bc_rec.b = new; error =xfs_btree_insert (cur, &i);if (error && error != -ENOSPC ) - goto done; +return error;/* * If get no-space back from btree insert, it tried a * split, and we have a zero block reservation. Fix up
@@ -5168,33 +5164,28 @@ xfs_bmap_del_extent_real(*/ error =xfs_bmbt_lookup_eq (cur, &got, &i);if (error) - goto done; -if (XFS_IS_CORRUPT (mp, i !=1 )) { - error = -EFSCORRUPTED ; - goto done; - } +return error; +if (XFS_IS_CORRUPT (mp, i !=1 )) +return -EFSCORRUPTED ;/* * Update the btree record back * to the original value. */ error =xfs_bmbt_update (cur, &old);if (error) - goto done; +return error;/* * Reset the extent record back * to the original value. */ xfs_iext_update_extent (ip, state, icur, &old); - flags =0 ; - error = -ENOSPC ; - goto done; - } -if (XFS_IS_CORRUPT (mp, i !=1 )) { - error = -EFSCORRUPTED ; - goto done; + *logflagsp =0 ; +return -ENOSPC ; } +if (XFS_IS_CORRUPT (mp, i !=1 )) +return -EFSCORRUPTED ; }else - flags |=xfs_ilog_fext (whichfork); + *logflagsp |=xfs_ilog_fext (whichfork); ifp->if_nextents++;xfs_iext_next (ifp, icur);
@@ -5218,7 +5209,7 @@ xfs_bmap_del_extent_real( ((bflags &XFS_BMAPI_NODISCARD ) || del->br_state == XFS_EXT_UNWRITTEN));if (error) - goto done; +return error; } }
@@ -5233,9 +5224,7 @@ xfs_bmap_del_extent_real(if (qfield && !(bflags &XFS_BMAPI_REMAP ))xfs_trans_mod_dquot_byino (tp, ip, qfield, (long )-nblks); -done: - *logflagsp = flags; -return error; +return 0 ; }/*
PATCH 22/67 xfs: update dir3 leaf block metadata after swap
In reply to Darrick J. WongFrom: Zhang Tianci <[email protected]>
Source kernel commit: 5759aa4f956034b289b0ae2c99daddfc775442e1
xfs_da3_swap_lastblock() copy the last block content to the dead block, but do not update the metadata in it. We need update some metadata for some kinds of type block, such as dir3 leafn block records its blkno, we shall update it to the dead block blkno. Otherwise, before write the xfs_buf to disk, the verify_write() will fail in blk_hdr->blkno != xfs_buf->b_bn, then xfs will be shutdown.
We will get this warning:
XFS (dm-0): Metadata corruption detected at xfs_dir3_leaf_verify+0xa8/0xe0 [xfs], xfs_dir3_leafn block 0x178 XFS (dm-0): Unmount and run xfs_repair
XFS (dm-0): First 128 bytes of corrupted metadata buffer:
00000000e80f1917: 00 80 00 0b 00 80 00 07 3d ff 00 00 00 00 00 00 ........=....... 000000009604c005: 00 00 00 00 00 00 01 a0 00 00 00 00 00 00 00 00 ................ 000000006b6fb2bf: e4 44 e3 97 b5 64 44 41 8b 84 60 0e 50 43 d9 bf .D...dDA..`.PC.. 00000000678978a2: 00 00 00 00 00 00 00 83 01 73 00 93 00 00 00 00 .........s...... 00000000b28b247c: 99 29 1d 38 00 00 00 00 99 29 1d 40 00 00 00 00 .).8.....).@.... 000000002b2a662c: 99 29 1d 48 00 00 00 00 99 49 11 00 00 00 00 00 .).H.....I...... 00000000ea2ffbb8: 99 49 11 08 00 00 45 25 99 49 11 10 00 00 48 fe .I....E%.I....H. 0000000069e86440: 99 49 11 18 00 00 4c 6b 99 49 11 20 00 00 4d 97 .I....Lk.I. ..M.
XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1423 of file fs/xfs/xfs_buf.c. Return address = 00000000c0ff63c1 XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem
XFS (dm-0): Please umount the filesystem and rectify the problem(s)
Quoted 1 line
From the log above, we know
xfs_buf->b_nois0x178, but the block's hdr record
its blkno is 0x1a0.
--- libxfs/xfs_da_btree.c | 7 +++++++ 1 file changed, 7 insertions(+)
libxfs/xfs_da_btree.c +7 -0
@@ -2312,10 +2312,17 @@ xfs_da3_swap_lastblock(return error;/* * Copy the last block into the dead buffer and log it. +* On CRC-enabled file systems, also update the stamped in blkno. */ memcpy (dead_buf->b_addr, last_buf->b_addr, args->geo->blksize); +if (xfs_has_crc (mp)) { +struct xfs_da3_blkinfo *da3 = dead_buf->b_addr; + + da3->blkno =cpu_to_be64 (xfs_buf_daddr (dead_buf)); + }xfs_trans_log_buf (tp, dead_buf,0 , args->geo->blksize -1 ); dead_info = dead_buf->b_addr; +/* * Get values from the moved block. */
PATCH 23/67 xfs: extract xfs_da_buf_copy() helper function
In reply to Darrick J. WongFrom: Zhang Tianci <[email protected]>
Source kernel commit: fd45ddb9dd606b3eaddf26e13f64340636955986
This patch does not modify logic.
xfs_da_buf_copy() will copy one block from src xfs_buf to dst xfs_buf, and update the block metadata in dst directly.
--- libxfs/xfs_attr_leaf.c | 12 +++----- libxfs/xfs_da_btree.c | 74 +++++++++++++++++++----------------------------- libxfs/xfs_da_btree.h | 2 + 3 files changed, 36 insertions(+), 52 deletions(-)
libxfs/xfs_attr_leaf.c +4 -8
@@ -1241,14 +1241,10 @@ xfs_attr3_leaf_to_node(if (error) goto out; -/* copy leaf to new buffer, update identifiers */ -xfs_trans_buf_set_type (args->trans, bp2,XFS_BLFT_ATTR_LEAF_BUF ); - bp2->b_ops = bp1->b_ops; -memcpy (bp2->b_addr, bp1->b_addr, args->geo->blksize); -if (xfs_has_crc (mp)) { -struct xfs_da3_blkinfo *hdr3 = bp2->b_addr; - hdr3->blkno =cpu_to_be64 (xfs_buf_daddr (bp2)); - } +/* +* Copy leaf to new buffer and log it. +*/ +xfs_da_buf_copy (bp2, bp1, args->geo->blksize);xfs_trans_log_buf (args->trans, bp2,0 , args->geo->blksize -1 );/*
libxfs/xfs_da_btree.c +30 -44
114 lines not on this page · raw message
libxfs/xfs_da_btree.h +2 -0
@@ -219,6 +219,8 @@ int xfs_da_reada_buf(struct xfs_inode *dp, xfs_dablk_t bno, const struct xfs_buf_ops *ops);int xfs_da_shrink_inode (xfs_da_args_t *args,xfs_dablk_t dead_blkno,struct xfs_buf *dead_buf); +void xfs_da_buf_copy (struct xfs_buf *dst,struct xfs_buf *src, +size_t size);uint xfs_da_hashname (const uint8_t *name_string,int name_length); enum xfs_dacmp xfs_da_compname(struct xfs_da_args *args,
PATCH 24/67 xfs: move xfs_ondisk.h to libxfs/
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Move xfs_ondisk.h to libxfs so that we can do the struct sanity checks in userspace libxfs as well. This should allow us to retire the somewhat fragile xfs/122 test on xfstests.
--- libxfs/init.c | 6 ++ libxfs/xfs_ondisk.h | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 205 insertions(+) create mode 100644 libxfs/xfs_ondisk.h
libxfs/init.c +6 -0
@@ -23,6 +23,11 @@#include "xfs_refcount_btree.h" #include "libfrog/platform.h" +#include "xfs_format.h" +#include "xfs_da_format.h" +#include "xfs_log_format.h" +#include "xfs_ondisk.h" +#include "libxfs.h" /* for now */ #ifndef HAVE_LIBURCU_ATOMIC64
@@ -248,6 +253,7 @@ libxfs_close_devices(int libxfs_init (struct libxfs_init *a) { +xfs_check_ondisk_structs ();rcu_init ();rcu_register_thread ();radix_tree_init ();
libxfs/xfs_ondisk.h new file +199 -0
199 lines not on this page · raw message
PATCH 25/67 xfs: consolidate the xfs_attr_defer_* helpers
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: c00eebd09e95757c9c1d08f0a6bbc32c543daf90
Consolidate the xfs_attr_defer_* helpers into a single xfs_attr_defer_add one that picks the right dela_state based on the passed in operation. Also move to a single trace point as the actual operation is visible through the flags in the delta_state passed to the trace point.
--- libxfs/xfs_attr.c | 90 ++++++++++++----------------------------------------- 1 file changed, 21 insertions(+), 69 deletions(-)
libxfs/xfs_attr.c +21 -69
@@ -878,11 +878,10 @@ xfs_attr_lookup(return error; } -static int -xfs_attr_intent_init ( +static void +xfs_attr_defer_add (struct xfs_da_args *args, -unsigned int op_flags,/* op flag (set or remove) */ -struct xfs_attr_intent **attr)/* new xfs_attr_intent */ +unsigned int op_flags) {struct xfs_attr_intent *new;
@@ -891,66 +890,22 @@ xfs_attr_intent_init( new->xattri_op_flags = op_flags; new->xattri_da_args = args; - *attr = new; -return 0 ; -} +switch (op_flags) { +case XFS_ATTRI_OP_FLAGS_SET : + new->xattri_dela_state =xfs_attr_init_add_state (args); +break ; +case XFS_ATTRI_OP_FLAGS_REPLACE : + new->xattri_dela_state =xfs_attr_init_replace_state (args); +break ; +case XFS_ATTRI_OP_FLAGS_REMOVE : + new->xattri_dela_state =xfs_attr_init_remove_state (args); +break ; +default : +ASSERT (0 ); + } -/* Sets an attribute for an inode as a deferred operation */ -static int -xfs_attr_defer_add ( -struct xfs_da_args *args) -{ -struct xfs_attr_intent *new; -int error =0 ; - - error =xfs_attr_intent_init (args,XFS_ATTRI_OP_FLAGS_SET , &new); -if (error) -return error; - - new->xattri_dela_state =xfs_attr_init_add_state (args);xfs_defer_add (args->trans,XFS_DEFER_OPS_TYPE_ATTR , &new->xattri_list);trace_xfs_attr_defer_add (new->xattri_dela_state, args->dp); - -return 0 ; -} - -/* Sets an attribute for an inode as a deferred operation */ -static int -xfs_attr_defer_replace ( -struct xfs_da_args *args) -{ -struct xfs_attr_intent *new; -int error =0 ; - - error =xfs_attr_intent_init (args,XFS_ATTRI_OP_FLAGS_REPLACE , &new); -if (error) -return error; - - new->xattri_dela_state =xfs_attr_init_replace_state (args); -xfs_defer_add (args->trans,XFS_DEFER_OPS_TYPE_ATTR , &new->xattri_list); -trace_xfs_attr_defer_replace (new->xattri_dela_state, args->dp); - -return 0 ; -} - -/* Removes an attribute for an inode as a deferred operation */ -static int -xfs_attr_defer_remove ( -struct xfs_da_args *args) -{ - -struct xfs_attr_intent *new; -int error; - - error =xfs_attr_intent_init (args,XFS_ATTRI_OP_FLAGS_REMOVE , &new); -if (error) -return error; - - new->xattri_dela_state =xfs_attr_init_remove_state (args); -xfs_defer_add (args->trans,XFS_DEFER_OPS_TYPE_ATTR , &new->xattri_list); -trace_xfs_attr_defer_remove (new->xattri_dela_state, args->dp); - -return 0 ; }/*
@@ -1036,16 +991,16 @@ xfs_attr_set( error =xfs_attr_lookup (args);switch (error) {case -EEXIST : -/* if no value, we are performing a remove operation */ if (!args->value) { - error =xfs_attr_defer_remove (args); +/* if no value, we are performing a remove operation */ +xfs_attr_defer_add (args,XFS_ATTRI_OP_FLAGS_REMOVE );break ; } +/* Pure create fails if the attr already exists */ if (args->attr_flags &XATTR_CREATE ) goto out_trans_cancel; - - error =xfs_attr_defer_replace (args); +xfs_attr_defer_add (args,XFS_ATTRI_OP_FLAGS_REPLACE );break ;case -ENOATTR :/* Can't remove what isn't there. */
@@ -1055,14 +1010,11 @@ xfs_attr_set(/* Pure replace fails if no existing attr to replace. */ if (args->attr_flags &XATTR_REPLACE ) goto out_trans_cancel; - - error =xfs_attr_defer_add (args); +xfs_attr_defer_add (args,XFS_ATTRI_OP_FLAGS_SET );break ;default : goto out_trans_cancel; } -if (error) - goto out_trans_cancel;/* * If this is a synchronous mount, make sure that the
PATCH 26/67 xfs: store an ops pointer in struct xfs_defer_pending
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 7f2f7531e0d455f1abb9f48fbbe17c37e8742590
The dfp_type field in struct xfs_defer_pending is only used to either look up the operations associated with the pending word or in trace points. Replace it with a direct pointer to the operations vector, and store a pretty name in the vector for tracing.
--- libxfs/defer_item.c | 6 ++++++ libxfs/xfs_defer.c | 43 +++++++++++++++++-------------------------- libxfs/xfs_defer.h | 5 +++-- 3 files changed, 26 insertions(+), 28 deletions(-)
libxfs/defer_item.c +6 -0
@@ -152,6 +152,7 @@ xfs_extent_free_cancel_item( }const struct xfs_defer_op_type xfs_extent_free_defer_type = { + .name ="extent_free" , .create_intent = xfs_extent_free_create_intent, .abort_intent = xfs_extent_free_abort_intent, .create_done = xfs_extent_free_create_done,
@@ -195,6 +196,7 @@ xfs_agfl_free_finish_item(/* sub-type with special handling for AGFL deferred frees */ const struct xfs_defer_op_type xfs_agfl_free_defer_type = { + .name ="agfl_free" , .create_intent = xfs_extent_free_create_intent, .abort_intent = xfs_extent_free_abort_intent, .create_done = xfs_extent_free_create_done,
@@ -306,6 +308,7 @@ xfs_rmap_update_cancel_item( }const struct xfs_defer_op_type xfs_rmap_update_defer_type = { + .name ="rmap" , .create_intent = xfs_rmap_update_create_intent, .abort_intent = xfs_rmap_update_abort_intent, .create_done = xfs_rmap_update_create_done,
@@ -424,6 +427,7 @@ xfs_refcount_update_cancel_item( }const struct xfs_defer_op_type xfs_refcount_update_defer_type = { + .name ="refcount" , .create_intent = xfs_refcount_update_create_intent, .abort_intent = xfs_refcount_update_abort_intent, .create_done = xfs_refcount_update_create_done,
@@ -546,6 +550,7 @@ xfs_bmap_update_cancel_item( }const struct xfs_defer_op_type xfs_bmap_update_defer_type = { + .name ="bmap" , .create_intent = xfs_bmap_update_create_intent, .abort_intent = xfs_bmap_update_abort_intent, .create_done = xfs_bmap_update_create_done,
@@ -641,6 +646,7 @@ xfs_attr_cancel_item( }const struct xfs_defer_op_type xfs_attr_defer_type = { + .name ="attr" , .max_items =1 , .create_intent = xfs_attr_create_intent, .abort_intent = xfs_attr_abort_intent,
libxfs/xfs_defer.c +17 -26
161 lines not on this page · raw message
libxfs/xfs_defer.h +3 -2
@@ -34,9 +34,9 @@ struct xfs_defer_pending {struct list_head dfp_work;/* work items */ struct xfs_log_item *dfp_intent;/* log intent item */ struct xfs_log_item *dfp_done;/* log done item */ +const struct xfs_defer_op_type *dfp_ops;unsigned int dfp_count;/* # extent items */ unsigned int dfp_flags; -enum xfs_defer_ops_type dfp_type; };/*
@@ -61,6 +61,8 @@ void xfs_defer_move(struct xfs_trans *dtp, struct xfs_trans *stp);/* Description of a deferred type. */ struct xfs_defer_op_type { +const char *name; +unsigned int max_items;struct xfs_log_item *(*create_intent)(struct xfs_trans *tp,struct list_head *items,unsigned int count,bool sort);void (*abort_intent)(struct xfs_log_item *intent);
@@ -76,7 +78,6 @@ struct xfs_defer_op_type {struct xfs_log_item *(*relog_intent)(struct xfs_trans *tp,struct xfs_log_item *intent,struct xfs_log_item *done_item); -unsigned int max_items; };extern const struct xfs_defer_op_type xfs_bmap_update_defer_type;
PATCH 27/67 xfs: pass the defer ops instead of type to xfs_defer_start_recovery
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: dc22af64368291a86fb6b7eb2adab21c815836b7
xfs_defer_start_recovery is only called from xlog_recover_intent_item, and the callers of that all have the actual xfs_defer_ops_type operation vector at hand. Pass that directly instead of looking it up from the defer_op_types table.
--- libxfs/xfs_defer.c | 6 +++--- libxfs/xfs_defer.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
libxfs/xfs_defer.c +3 -3
@@ -888,14 +888,14 @@ xfs_defer_add_barrier(void xfs_defer_start_recovery (struct xfs_log_item *lip, -enum xfs_defer_ops_type dfp_type, -struct list_head *r_dfops) +struct list_head *r_dfops, +const struct xfs_defer_op_type *ops) {struct xfs_defer_pending *dfp; dfp =kmem_cache_zalloc (xfs_defer_pending_cache,GFP_NOFS | __GFP_NOFAIL); - dfp->dfp_ops = defer_op_types[dfp_type]; + dfp->dfp_ops = ops; dfp->dfp_intent = lip;INIT_LIST_HEAD (&dfp->dfp_work);list_add_tail (&dfp->dfp_list, r_dfops);
libxfs/xfs_defer.h +1 -1
@@ -147,7 +147,7 @@ void xfs_defer_ops_capture_abort(struct xfs_mount *mp,void xfs_defer_resources_rele (struct xfs_defer_resources *dres);void xfs_defer_start_recovery (struct xfs_log_item *lip, -enum xfs_defer_ops_type dfp_type,struct list_head *r_dfops); +struct list_head *r_dfops,const struct xfs_defer_op_type *ops);void xfs_defer_cancel_recovery (struct xfs_mount *mp,struct xfs_defer_pending *dfp); int xfs_defer_finish_recovery(struct xfs_mount *mp,
PATCH 28/67 xfs: pass the defer ops directly to xfs_defer_add
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 603ce8ab12094a2d9483c79a7541335e258a5328
Pass a pointer to the xfs_defer_op_type structure to xfs_defer_add and remove the indirection through the xfs_defer_ops_type enum and a global table of all possible operations.
--- libxfs/xfs_alloc.c | 4 ++-- libxfs/xfs_attr.c | 2 +- libxfs/xfs_bmap.c | 2 +- libxfs/xfs_defer.c | 16 ++-------------- libxfs/xfs_defer.h | 18 ++---------------- libxfs/xfs_refcount.c | 2 +- libxfs/xfs_rmap.c | 2 +- 7 files changed, 10 insertions(+), 36 deletions(-)
libxfs/xfs_alloc.c +2 -2
@@ -2510,7 +2510,7 @@ xfs_defer_agfl_block(trace_xfs_agfl_free_defer (mp, agno,0 , agbno,1 );xfs_extent_free_get_group (mp, xefi); -xfs_defer_add (tp,XFS_DEFER_OPS_TYPE_AGFL_FREE , &xefi->xefi_list); +xfs_defer_add (tp, &xefi->xefi_list, &xfs_agfl_free_defer_type);return 0 ; }
@@ -2574,7 +2574,7 @@ xfs_defer_extent_free( XFS_FSB_TO_AGBNO(tp->t_mountp, bno), len);xfs_extent_free_get_group (mp, xefi); - *dfpp =xfs_defer_add (tp,XFS_DEFER_OPS_TYPE_FREE , &xefi->xefi_list); + *dfpp =xfs_defer_add (tp, &xefi->xefi_list, &xfs_extent_free_defer_type);return 0 ; }
libxfs/xfs_attr.c +1 -1
@@ -904,7 +904,7 @@ xfs_attr_defer_add(ASSERT (0 ); } -xfs_defer_add (args->trans,XFS_DEFER_OPS_TYPE_ATTR , &new->xattri_list); +xfs_defer_add (args->trans, &new->xattri_list, &xfs_attr_defer_type);trace_xfs_attr_defer_add (new->xattri_dela_state, args->dp); }
libxfs/xfs_bmap.c +1 -1
@@ -6085,7 +6085,7 @@ __xfs_bmap_add( bi->bi_bmap = *bmap;xfs_bmap_update_get_group (tp->t_mountp, bi); -xfs_defer_add (tp,XFS_DEFER_OPS_TYPE_BMAP , &bi->bi_list); +xfs_defer_add (tp, &bi->bi_list, &xfs_bmap_update_defer_type);return 0 ; }
libxfs/xfs_defer.c +2 -14
@@ -229,16 +229,6 @@ static const struct xfs_defer_op_type xfs_barrier_defer_type = { .cancel_item = xfs_defer_barrier_cancel_item, }; -static const struct xfs_defer_op_type *defer_op_types[] = { - [XFS_DEFER_OPS_TYPE_BMAP ] = &xfs_bmap_update_defer_type, - [XFS_DEFER_OPS_TYPE_REFCOUNT ] = &xfs_refcount_update_defer_type, - [XFS_DEFER_OPS_TYPE_RMAP ] = &xfs_rmap_update_defer_type, - [XFS_DEFER_OPS_TYPE_FREE ] = &xfs_extent_free_defer_type, - [XFS_DEFER_OPS_TYPE_AGFL_FREE ] = &xfs_agfl_free_defer_type, - [XFS_DEFER_OPS_TYPE_ATTR ] = &xfs_attr_defer_type, - [XFS_DEFER_OPS_TYPE_BARRIER ] = &xfs_barrier_defer_type, -}; -/* Create a log intent done item for a log intent item. */ static inline void xfs_defer_create_done(
@@ -841,14 +831,12 @@ xfs_defer_alloc(struct xfs_defer_pending *xfs_defer_add (struct xfs_trans *tp, -enum xfs_defer_ops_type type, -struct list_head *li) +struct list_head *li, +const struct xfs_defer_op_type *ops) {struct xfs_defer_pending *dfp =NULL ; -const struct xfs_defer_op_type *ops = defer_op_types[type];ASSERT (tp->t_flags &XFS_TRANS_PERM_LOG_RES ); -BUILD_BUG_ON (ARRAY_SIZE (defer_op_types) !=XFS_DEFER_OPS_TYPE_MAX ); dfp =xfs_defer_find_last (tp, ops);if (!dfp || !xfs_defer_can_append (dfp, ops))
libxfs/xfs_defer.h +2 -16
@@ -10,20 +10,6 @@ struct xfs_btree_cur;struct xfs_defer_op_type ;struct xfs_defer_capture ; -/* -* Header for deferred operation list. -*/ -enum xfs_defer_ops_type { -XFS_DEFER_OPS_TYPE_BMAP , -XFS_DEFER_OPS_TYPE_REFCOUNT , -XFS_DEFER_OPS_TYPE_RMAP , -XFS_DEFER_OPS_TYPE_FREE , -XFS_DEFER_OPS_TYPE_AGFL_FREE , -XFS_DEFER_OPS_TYPE_ATTR , -XFS_DEFER_OPS_TYPE_BARRIER , -XFS_DEFER_OPS_TYPE_MAX , -}; -/* * Save a log intent item and a list of extents, so that we can replay * whatever action had to happen to the extent list and file the log done
@@ -51,8 +37,8 @@ struct xfs_defer_pending {void xfs_defer_item_pause (struct xfs_trans *tp,struct xfs_defer_pending *dfp);void xfs_defer_item_unpause (struct xfs_trans *tp,struct xfs_defer_pending *dfp); -struct xfs_defer_pending *xfs_defer_add (struct xfs_trans *tp, -enum xfs_defer_ops_type type,struct list_head *h); +struct xfs_defer_pending *xfs_defer_add (struct xfs_trans *tp,struct list_head *h, +const struct xfs_defer_op_type *ops);int xfs_defer_finish_noroll (struct xfs_trans **tp);int xfs_defer_finish (struct xfs_trans **tp);int xfs_defer_finish_one (struct xfs_trans *tp,struct xfs_defer_pending *dfp);
libxfs/xfs_refcount.c +1 -1
@@ -1457,7 +1457,7 @@ __xfs_refcount_add( ri->ri_blockcount = blockcount;xfs_refcount_update_get_group (tp->t_mountp, ri); -xfs_defer_add (tp,XFS_DEFER_OPS_TYPE_REFCOUNT , &ri->ri_list); +xfs_defer_add (tp, &ri->ri_list, &xfs_refcount_update_defer_type); }/*
libxfs/xfs_rmap.c +1 -1
@@ -2566,7 +2566,7 @@ __xfs_rmap_add( ri->ri_bmap = *bmap;xfs_rmap_update_get_group (tp->t_mountp, ri); -xfs_defer_add (tp,XFS_DEFER_OPS_TYPE_RMAP , &ri->ri_list); +xfs_defer_add (tp, &ri->ri_list, &xfs_rmap_update_defer_type); }/* Map an extent into a file. */
PATCH 29/67 xfs: force all buffers to be written during btree bulk load
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 13ae04d8d45227c2ba51e188daf9fc13d08a1b12
While stress-testing online repair of btrees, I noticed periodic assertion failures from the buffer cache about buffers with incorrect DELWRI_Q state. Looking further, I observed this race between the AIL trying to write out a btree block and repair zapping a btree block after the fact:
AIL: Repair0:
pin buffer Xdelwri_queue:
set DELWRI_Q
add to delwri list
stale buf X:
clear DELWRI_Q
does not clear b_list
free space X
delwri_submit # oops
Worse yet, I discovered that running the same repair over and over in a tight loop can result in a second race that cause data integrity problems with the repair:
AIL: Repair0: Repair1:
pin buffer Xdelwri_queue:
set DELWRI_Q
add to delwri list
stale buf X:
clear DELWRI_Q
does not clear b_list
free space X
find free space X
get buffer
rewrite bufferdelwri_queue:
set DELWRI_Q
already on a list, do not add
BAD: committed tree root before all blocks written
delwri_submit # too late now
I traced this to my own misunderstanding of how the delwri lists work, particularly with regards to the AIL's buffer list. If a buffer is logged and committed, the buffer can end up on that AIL buffer list. If btree repairs are run twice in rapid succession, it's possible that the first repair will invalidate the buffer and free it before the next time the AIL wakes up. Marking the buffer stale clears DELWRI_Q from the buffer state without removing the buffer from its delwri list. The buffer doesn't know which list it's on, so it cannot know which lock to take to protect the list for a removal.
If the second repair allocates the same block, it will then recycle the buffer to start writing the new btree block. Meanwhile, if the AIL wakes up and walks the buffer list, it will ignore the buffer because it can't lock it, and go back to sleep.
When the second repair calls delwri_queue to put the buffer on the list of buffers to write before committing the new btree, it will set DELWRI_Q again, but since the buffer hasn't been removed from the AIL's buffer list, it won't add it to the bulkload buffer's list.
This is incorrect, because the bulkload caller relies on delwri_submit to ensure that all the buffers have been sent to disk /before/ required for data consistency.
Worse, the AIL won't clear DELWRI_Q from the buffer when it does finally drop it, so the next thread to walk through the btree will trip over a debug assertion on that flag.
To fix this, create a new function that waits for the buffer to be removed from any other delwri lists before adding the buffer to the caller's delwri list. By waiting for the buffer to clear both the delwri list and any potential delwri wait list, we can be sure that repair will initiate writes of all buffers and report all write errors back to userspace instead of committing the new structure.
--- libxfs/libxfs_io.h | 11 +++++++++++ libxfs/xfs_btree_staging.c | 4 +--- 2 files changed, 12 insertions(+), 3 deletions(-)
libxfs/libxfs_io.h +11 -0
@@ -244,6 +244,17 @@ xfs_buf_delwri_queue(struct xfs_buf *bp, struct list_head *buffer_list)return true; } +static inline void +xfs_buf_delwri_queue_here (struct xfs_buf *bp,struct list_head *buffer_list) +{ +ASSERT (list_empty (&bp->b_list)); + +/* This buffer is uptodate; don't let it get reread. */ +libxfs_buf_mark_dirty (bp); + +xfs_buf_delwri_queue (bp, buffer_list); +} +int xfs_buf_delwri_submit (struct list_head *buffer_list);void xfs_buf_delwri_cancel (struct list_head *list);
libxfs/xfs_btree_staging.c +1 -3
@@ -342,9 +342,7 @@ xfs_btree_bload_drop_buf(if (*bpp ==NULL )return ; -if (!xfs_buf_delwri_queue (*bpp, buffers_list)) -ASSERT (0 ); - +xfs_buf_delwri_queue_here (*bpp, buffers_list);xfs_buf_relse (*bpp); *bpp =NULL ; }
PATCH 30/67 xfs: set XBF_DONE on newly formatted btree block that are ready for writing
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: c1e0f8e6fb060b23b6f1b82eb4265983f7d271f8
The btree bulkloading code calls xfs_buf_delwri_queue_here when it has finished formatting a new btree block and wants to queue it to be written to disk. Once the new btree root has been committed, the blocks (and hence the buffers) will be accessible to the rest of the filesystem. Mark each new buffer as DONE when adding it to the delwri list so that the next btree traversal can skip reloading the contents from disk.
--- libxfs/xfs_btree_staging.c | 6 ++++++ 1 file changed, 6 insertions(+)
libxfs/xfs_btree_staging.c +6 -0
@@ -342,6 +342,12 @@ xfs_btree_bload_drop_buf(if (*bpp ==NULL )return ; +/* +* Mark this buffer XBF_DONE (i.e. uptodate) so that a subsequent +* xfs_buf_read will not pointlessly reread the contents from the disk. +*/ + (*bpp)->b_flags |=XBF_DONE ; +xfs_buf_delwri_queue_here (*bpp, buffers_list);xfs_buf_relse (*bpp); *bpp =NULL ;
PATCH 31/67 xfs: read leaf blocks when computing keys for bulkloading into node blocks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 26de64629d8b439a03bce243f14a46f7440729f3
When constructing a new btree, xfs_btree_bload_node needs to read the btree blocks for level N to compute the keyptrs for the blocks that will be loaded into level N+1. The level N blocks must be formatted at that point.
A subsequent patch will change the btree bulkloader to write new btree blocks in 256K chunks to moderate memory consumption if the new btree is very large. As a consequence of that, it's possible that the buffers for lower level blocks might have been reclaimed by the time the node builder comes back to the block.
Therefore, change xfs_btree_bload_node to read the lower level blocks to handle the reclaimed buffer case. As a side effect, the read will increase the LRU refs, which will bias towards keeping new btree buffers in memory after the new btree commits.
--- libxfs/xfs_btree.c | 2 +- libxfs/xfs_btree.h | 3 +++ libxfs/xfs_btree_staging.c | 7 ++++++- 3 files changed, 10 insertions(+), 2 deletions(-)
libxfs/xfs_btree.c +1 -1
@@ -1327,7 +1327,7 @@ xfs_btree_get_buf_block(* Read in the buffer at the given ptr and return the buffer and * the block pointer within the buffer. */ -STATIC int +int xfs_btree_read_buf_block( struct xfs_btree_cur *cur, const union xfs_btree_ptr *ptr,
libxfs/xfs_btree.h +3 -0
@@ -700,6 +700,9 @@ void xfs_btree_set_ptr_null(struct xfs_btree_cur *cur,int xfs_btree_get_buf_block (struct xfs_btree_cur *cur,const union xfs_btree_ptr *ptr,struct xfs_btree_block **block,struct xfs_buf **bpp); +int xfs_btree_read_buf_block (struct xfs_btree_cur *cur, +const union xfs_btree_ptr *ptr,int flags, +struct xfs_btree_block **block,struct xfs_buf **bpp);void xfs_btree_set_sibling (struct xfs_btree_cur *cur,struct xfs_btree_block *block,const union xfs_btree_ptr *ptr,int lr);
libxfs/xfs_btree_staging.c +6 -1
@@ -489,7 +489,12 @@ xfs_btree_bload_node(ASSERT (!xfs_btree_ptr_is_null (cur, child_ptr)); - ret =xfs_btree_get_buf_block (cur, child_ptr, &child_block, +/* +* Read the lower-level block in case the buffer for it has +* been reclaimed. LRU refs will be set on the block, which is +* desirable if the new btree commits. +*/ + ret =xfs_btree_read_buf_block (cur, child_ptr,0 , &child_block, &child_bp);if (ret)return ret;
PATCH 32/67 xfs: move btree bulkload record initialization to ->get_record implementations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 6dfeb0c2ecde71d61af77f65eabbdd6ca9315161
When we're performing a bulk load of a btree, move the code that actually stores the btree record in the new btree block out of the generic code and into the individual ->get_record implementations. This is preparation for being able to store multiple records with a single indirect call.
--- libxfs/libxfs_api_defs.h | 1 + libxfs/xfs_btree_staging.c | 17 ++++++------- libxfs/xfs_btree_staging.h | 15 ++++++++---- repair/agbtree.c | 56 +++++++++++++++++++++++++++++++++----------- 4 files changed, 60 insertions(+), 29 deletions(-)
libxfs/libxfs_api_defs.h +1 -0
@@ -53,6 +53,7 @@#define xfs_btree_bload_compute_geometry libxfs_btree_bload_compute_geometry#define xfs_btree_del_cursor libxfs_btree_del_cursor#define xfs_btree_init_block libxfs_btree_init_block +#define xfs_btree_rec_addr libxfs_btree_rec_addr#define xfs_buf_delwri_submit libxfs_buf_delwri_submit#define xfs_buf_get libxfs_buf_get#define xfs_buf_get_uncached libxfs_buf_get_uncached
libxfs/xfs_btree_staging.c +7 -10
@@ -440,22 +440,19 @@ STATIC intxfs_btree_bload_leaf ( struct xfs_btree_cur *cur, unsigned int recs_this_block, - xfs_btree_bload_get_record_fn get_record, + xfs_btree_bload_get_records_fn get_records, struct xfs_btree_block *block, void *priv) { - unsigned int j; + unsigned int j = 1;int ret;/* Fill the leaf block with records. */ -for (j =1 ; j <= recs_this_block; j++) { -union xfs_btree_rec *block_rec; - - ret =get_record (cur, priv); -if (ret) +while (j <= recs_this_block) { + ret =get_records (cur, j, block, recs_this_block - j +1 , priv); +if (ret <0 )return ret; - block_rec =xfs_btree_rec_addr (cur, j, block); - cur->bc_ops->init_rec_from_cur (cur, block_rec); + j += ret; }return 0 ;
@@ -798,7 +795,7 @@ xfs_btree_bload(trace_xfs_btree_bload_block (cur, level, i, blocks, &ptr, nr_this_block); - ret =xfs_btree_bload_leaf (cur, nr_this_block, bbl->get_record, + ret =xfs_btree_bload_leaf (cur, nr_this_block, bbl->get_records, block, priv);if (ret) goto out;
libxfs/xfs_btree_staging.h +10 -5
@@ -47,7 +47,9 @@ void xfs_btree_commit_ifakeroot(struct xfs_btree_cur *cur, struct xfs_trans *tp, int whichfork, const struct xfs_btree_ops *ops);/* Bulk loading of staged btrees. */ -typedef int (*xfs_btree_bload_get_record_fn )(struct xfs_btree_cur *cur,void *priv); +typedef int (*xfs_btree_bload_get_records_fn )(struct xfs_btree_cur *cur, +unsigned int idx,struct xfs_btree_block *block, +unsigned int nr_wanted,void *priv);typedef int (*xfs_btree_bload_claim_block_fn )(struct xfs_btree_cur *cur,union xfs_btree_ptr *ptr,void *priv); typedef size_t (*xfs_btree_bload_iroot_size_fn)(struct xfs_btree_cur *cur,
@@ -55,11 +57,14 @@ typedef size_t (*xfs_btree_bload_iroot_size_fn)(struct xfs_btree_cur *cur,struct xfs_btree_bload {/* -* This function will be called nr_records times to load records into -* the btree. The function does this by setting the cursor's bc_rec -* field in in-core format. Records must be returned in sort order. +* This function will be called to load @nr_wanted records into the +* btree. The implementation does this by setting the cursor's bc_rec +* field in in-core format and using init_rec_from_cur to set the +* records in the btree block. Records must be returned in sort order. +* The function must return the number of records loaded or the usual +* negative errno. */ -xfs_btree_bload_get_record_fn get_record; +xfs_btree_bload_get_records_fn get_records;/* * This function will be called nr_blocks times to obtain a pointer
repair/agbtree.c +42 -14
150 lines not on this page · raw message
PATCH 33/67 xfs: constrain dirty buffers while formatting a staged btree
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: e069d549705e49841247acf9b3176744e27d5425
Constrain the number of dirty buffers that are locked by the btree staging code at any given time by establishing a threshold at which we put them all on the delwri queue and push them to disk. This limits memory consumption while writing out new btrees.
--- libxfs/xfs_btree_staging.c | 50 ++++++++++++++++++++++++++++++++++---------- libxfs/xfs_btree_staging.h | 10 +++++++++ repair/agbtree.c | 1 + 3 files changed, 50 insertions(+), 11 deletions(-)
libxfs/xfs_btree_staging.c +39 -11
@@ -333,24 +333,41 @@ xfs_btree_commit_ifakeroot(/* * Put a btree block that we're loading onto the ordered list and release it. * The btree blocks will be written to disk when bulk loading is finished. +* If we reach the dirty buffer threshold, flush them to disk before +* continuing. */ -static void +static int xfs_btree_bload_drop_buf ( -struct list_head *buffers_list, -struct xfs_buf **bpp) +struct xfs_btree_bload *bbl, +struct list_head *buffers_list, +struct xfs_buf **bpp) { -if (*bpp ==NULL ) -return ; +struct xfs_buf *bp = *bpp; +int error; + +if (!bp) +return 0 ;/* * Mark this buffer XBF_DONE (i.e. uptodate) so that a subsequent * xfs_buf_read will not pointlessly reread the contents from the disk. */ - (*bpp)->b_flags |=XBF_DONE ; + bp->b_flags |=XBF_DONE ; -xfs_buf_delwri_queue_here (*bpp, buffers_list); -xfs_buf_relse (*bpp); +xfs_buf_delwri_queue_here (bp, buffers_list); +xfs_buf_relse (bp); *bpp =NULL ; + bbl->nr_dirty++; + +if (!bbl->max_dirty || bbl->nr_dirty < bbl->max_dirty) +return 0 ; + + error =xfs_buf_delwri_submit (buffers_list); +if (error) +return error; + + bbl->nr_dirty =0 ; +return 0 ; }/*
@@ -422,7 +439,10 @@ xfs_btree_bload_prep_block(*/ if (*blockp)xfs_btree_set_sibling (cur, *blockp, &new_ptr,XFS_BB_RIGHTSIB ); -xfs_btree_bload_drop_buf (buffers_list, bpp); + + ret =xfs_btree_bload_drop_buf (bbl, buffers_list, bpp); +if (ret) +return ret;/* Initialize the new btree block. */ xfs_btree_init_block_cur (cur, new_bp, level, nr_this_block);
@@ -770,6 +790,7 @@ xfs_btree_bload( cur->bc_nlevels = bbl->btree_height;xfs_btree_set_ptr_null (cur, &child_ptr);xfs_btree_set_ptr_null (cur, &ptr); + bbl->nr_dirty =0 ;xfs_btree_bload_level_geometry (cur, bbl, level, nr_this_level, &avg_per_block, &blocks, &blocks_with_extra);
@@ -808,7 +829,10 @@ xfs_btree_bload(xfs_btree_copy_ptrs (cur, &child_ptr, &ptr,1 ); } total_blocks += blocks; -xfs_btree_bload_drop_buf (&buffers_list, &bp); + + ret =xfs_btree_bload_drop_buf (bbl, &buffers_list, &bp); +if (ret) + goto out;/* Populate the internal btree nodes. */ for (level =1 ; level < cur->bc_nlevels; level++) {
@@ -850,7 +874,11 @@ xfs_btree_bload(xfs_btree_copy_ptrs (cur, &first_ptr, &ptr,1 ); } total_blocks += blocks; -xfs_btree_bload_drop_buf (&buffers_list, &bp); + + ret =xfs_btree_bload_drop_buf (bbl, &buffers_list, &bp); +if (ret) + goto out; +xfs_btree_copy_ptrs (cur, &child_ptr, &first_ptr,1 ); }
libxfs/xfs_btree_staging.h +10 -0
@@ -112,6 +112,16 @@ struct xfs_btree_bload {* height of the new btree. */ unsigned int btree_height; + +/* +* Flush the new btree block buffer list to disk after this many blocks +* have been formatted. Zero prohibits writing any buffers until all +* blocks have been formatted. +*/ +uint16_t max_dirty; + +/* Number of dirty buffers. */ +uint16_t nr_dirty; }; int xfs_btree_bload_compute_geometry(struct xfs_btree_cur *cur,
repair/agbtree.c +1 -0
@@ -23,6 +23,7 @@ init_rebuild(memset (btr,0 ,sizeof (struct bt_rebuild ));bulkload_init_ag (&btr->newbt, sc, oinfo); + btr->bload.max_dirty =XFS_B_TO_FSBT (sc->mp,256U <<10 );/* 256K */ bulkload_estimate_ag_slack (sc, &btr->bload, est_agfreeblocks); }
PATCH 34/67 xfs: repair free space btrees
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 4bdfd7d15747b170ce93a06fafccaf20544b6684
Rebuild the free space btrees from the gaps in the rmap btree. Refer to the case study in Documentation/filesystems/xfs-online-fsck-design.rst for more details.
--- libxfs/xfs_ag.h | 9 +++++++++ libxfs/xfs_ag_resv.c | 2 ++ libxfs/xfs_alloc.c | 10 ++++------ libxfs/xfs_alloc.h | 2 +- libxfs/xfs_alloc_btree.c | 13 ++++++++++++- libxfs/xfs_types.h | 7 +++++++ 6 files changed, 35 insertions(+), 8 deletions(-)
libxfs/xfs_ag.h +9 -0
@@ -80,6 +80,15 @@ struct xfs_perag {*/ uint16_t pag_checked;uint16_t pag_sick; + +#ifdef CONFIG_XFS_ONLINE_REPAIR +/* +* Alternate btree heights so that online repair won't trip the write +* verifiers while rebuilding the AG btrees. +*/ +uint8_t pagf_repair_levels[XFS_BTNUM_AGF ]; +#endif +spinlock_t pag_state_lock;spinlock_t pagb_lock;/* lock for pagb_tree */
libxfs/xfs_ag_resv.c +2 -0
@@ -410,6 +410,8 @@ xfs_ag_resv_free_extent( fallthrough;case XFS_AG_RESV_NONE :xfs_trans_mod_sb (tp,XFS_TRANS_SB_FDBLOCKS , (int64_t )len); + fallthrough; +case XFS_AG_RESV_IGNORE :return ; }
libxfs/xfs_alloc.c +4 -6
@@ -242,11 +242,9 @@ xfs_alloc_btrec_to_irec(/* Simple checks for free space records. */ xfs_failaddr_t xfs_alloc_check_irec ( -struct xfs_btree_cur *cur, -const struct xfs_alloc_rec_incore *irec) +struct xfs_perag *pag, +const struct xfs_alloc_rec_incore *irec) { -struct xfs_perag *pag = cur->bc_ag.pag; -if (irec->ar_blockcount ==0 )return __this_address;
@@ -295,7 +293,7 @@ xfs_alloc_get_rec(return error;xfs_alloc_btrec_to_irec (rec, &irec); - fa =xfs_alloc_check_irec (cur, &irec); + fa =xfs_alloc_check_irec (cur->bc_ag.pag, &irec);if (fa)return xfs_alloc_complain_bad_rec (cur, fa, &irec);
@@ -3940,7 +3938,7 @@ xfs_alloc_query_range_helper(xfs_failaddr_t fa;xfs_alloc_btrec_to_irec (rec, &irec); - fa =xfs_alloc_check_irec (cur, &irec); + fa =xfs_alloc_check_irec (cur->bc_ag.pag, &irec);if (fa)return xfs_alloc_complain_bad_rec (cur, fa, &irec);
libxfs/xfs_alloc.h +1 -1
@@ -185,7 +185,7 @@ xfs_alloc_get_rec(union xfs_btree_rec ;void xfs_alloc_btrec_to_irec (const union xfs_btree_rec *rec,struct xfs_alloc_rec_incore *irec); -xfs_failaddr_t xfs_alloc_check_irec (struct xfs_btree_cur *cur, +xfs_failaddr_t xfs_alloc_check_irec (struct xfs_perag *pag,const struct xfs_alloc_rec_incore *irec); int xfs_read_agf(struct xfs_perag *pag, struct xfs_trans *tp, int flags,
libxfs/xfs_alloc_btree.c +12 -1
@@ -321,7 +321,18 @@ xfs_allocbt_verify(if (bp->b_ops->magic[0 ] ==cpu_to_be32 (XFS_ABTC_MAGIC )) btnum = XFS_BTNUM_CNTi;if (pag &&xfs_perag_initialised_agf (pag)) { -if (level >= pag->pagf_levels[btnum]) +unsigned int maxlevel = pag->pagf_levels[btnum]; + +#ifdef CONFIG_XFS_ONLINE_REPAIR +/* +* Online repair could be rewriting the free space btrees, so +* we'll validate against the larger of either tree while this +* is going on. +*/ + maxlevel = max_t(unsigned int, maxlevel, + pag->pagf_repair_levels[btnum]); +#endif +if (level >= maxlevel)return __this_address; }else if (level >= mp->m_alloc_maxlevels)return __this_address;
libxfs/xfs_types.h +7 -0
@@ -208,6 +208,13 @@ enum xfs_ag_resv_type {XFS_AG_RESV_AGFL ,XFS_AG_RESV_METADATA ,XFS_AG_RESV_RMAPBT , + +/* +* Don't increase fdblocks when freeing extent. This is a pony for +* the bnobt repair functions to re-free the free space without +* altering fdblocks. If you think you need this you're wrong. +*/ + XFS_AG_RESV_IGNORE, };/* Results of scanning a btree keyspace to check occupancy. */
PATCH 35/67 xfs: repair inode btrees
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: dbfbf3bdf639a20da7d5fb390cd2e197d25aa418
Use the rmapbt to find inode chunks, query the chunks to compute hole and free masks, and with that information rebuild the inobt and finobt. Refer to the case study in Documentation/filesystems/xfs-online-fsck-design.rst for more details.
--- libxfs/xfs_ialloc.c | 31 ++++++++++++++++++------------- libxfs/xfs_ialloc.h | 3 ++- 2 files changed, 20 insertions(+), 14 deletions(-)
libxfs/xfs_ialloc.c +18 -13
@@ -90,18 +90,28 @@ xfs_inobt_btrec_to_irec( irec->ir_free =be64_to_cpu (rec->inobt.ir_free); } +/* Compute the freecount of an incore inode record. */ +uint8_t +xfs_inobt_rec_freecount ( +const struct xfs_inobt_rec_incore *irec) +{ +uint64_t realfree = irec->ir_free; + +if (xfs_inobt_issparse (irec->ir_holemask)) + realfree &=xfs_inobt_irec_to_allocmask (irec); +return hweight64 (realfree); +} +/* Simple checks for inode records. */ xfs_failaddr_t xfs_inobt_check_irec ( -struct xfs_btree_cur *cur, +struct xfs_perag *pag,const struct xfs_inobt_rec_incore *irec) { -uint64_t realfree; -/* Record has to be properly aligned within the AG. */ -if (!xfs_verify_agino (cur->bc_ag.pag, irec->ir_startino)) +if (!xfs_verify_agino (pag, irec->ir_startino))return __this_address; -if (!xfs_verify_agino (cur->bc_ag.pag, +if (!xfs_verify_agino (pag, irec->ir_startino +XFS_INODES_PER_CHUNK -1 ))return __this_address; if (irec->ir_count < XFS_INODES_PER_HOLEMASK_BIT ||
@@ -110,12 +120,7 @@ xfs_inobt_check_irec(if (irec->ir_freecount >XFS_INODES_PER_CHUNK )return __this_address; -/* if there are no holes, return the first available offset */ -if (!xfs_inobt_issparse (irec->ir_holemask)) - realfree = irec->ir_free; -else - realfree = irec->ir_free &xfs_inobt_irec_to_allocmask (irec); -if (hweight64 (realfree) != irec->ir_freecount) +if (xfs_inobt_rec_freecount (irec) != irec->ir_freecount)return __this_address;return NULL ;
@@ -159,7 +164,7 @@ xfs_inobt_get_rec(return error;xfs_inobt_btrec_to_irec (mp, rec, irec); - fa =xfs_inobt_check_irec (cur, irec); + fa =xfs_inobt_check_irec (cur->bc_ag.pag, irec);if (fa)return xfs_inobt_complain_bad_rec (cur, fa, irec);
@@ -2735,7 +2740,7 @@ xfs_ialloc_count_inodes_rec(xfs_failaddr_t fa;xfs_inobt_btrec_to_irec (cur->bc_mp, rec, &irec); - fa =xfs_inobt_check_irec (cur, &irec); + fa =xfs_inobt_check_irec (cur->bc_ag.pag, &irec);if (fa)return xfs_inobt_complain_bad_rec (cur, fa, &irec);
libxfs/xfs_ialloc.h +2 -1
@@ -79,6 +79,7 @@ int xfs_inobt_lookup(struct xfs_btree_cur *cur, xfs_agino_t ino,*/ int xfs_inobt_get_rec (struct xfs_btree_cur *cur,xfs_inobt_rec_incore_t *rec,int *stat); +uint8_t xfs_inobt_rec_freecount (const struct xfs_inobt_rec_incore *irec);/* * Inode chunk initialisation routine
@@ -93,7 +94,7 @@ union xfs_btree_rec;void xfs_inobt_btrec_to_irec (struct xfs_mount *mp,const union xfs_btree_rec *rec,struct xfs_inobt_rec_incore *irec); -xfs_failaddr_t xfs_inobt_check_irec (struct xfs_btree_cur *cur, +xfs_failaddr_t xfs_inobt_check_irec (struct xfs_perag *pag,const struct xfs_inobt_rec_incore *irec); int xfs_ialloc_has_inodes_at_extent(struct xfs_btree_cur *cur, xfs_agblock_t bno, xfs_extlen_t len,
PATCH 36/67 xfs: repair refcount btrees
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 9099cd38002f8029c9a1da08e6832d1cd18e8451
Reconstruct the refcount data from the rmap btree.
--- libxfs/xfs_ag.h | 1 + libxfs/xfs_btree.c | 26 ++++++++++++++++++++++++++ libxfs/xfs_btree.h | 2 ++ libxfs/xfs_refcount.c | 8 +++----- libxfs/xfs_refcount.h | 2 +- libxfs/xfs_refcount_btree.c | 13 ++++++++++++- 6 files changed, 45 insertions(+), 7 deletions(-)
libxfs/xfs_ag.h +1 -0
@@ -87,6 +87,7 @@ struct xfs_perag {* verifiers while rebuilding the AG btrees. */ uint8_t pagf_repair_levels[XFS_BTNUM_AGF ]; +uint8_t pagf_repair_refcount_level;#endif spinlock_t pag_state_lock;
libxfs/xfs_btree.c +26 -0
@@ -5209,3 +5209,29 @@ xfs_btree_destroy_cur_caches(void)xfs_rmapbt_destroy_cur_cache ();xfs_refcountbt_destroy_cur_cache (); } + +/* Move the btree cursor before the first record. */ +int +xfs_btree_goto_left_edge ( +struct xfs_btree_cur *cur) +{ +int stat =0 ; +int error; + +memset (&cur->bc_rec,0 ,sizeof (cur->bc_rec)); + error =xfs_btree_lookup (cur,XFS_LOOKUP_LE , &stat); +if (error) +return error; +if (!stat) +return 0 ; + + error =xfs_btree_decrement (cur,0 , &stat); +if (error) +return error; +if (stat !=0 ) { +ASSERT (0 ); +return -EFSCORRUPTED ; + } + +return 0 ; +}
libxfs/xfs_btree.h +2 -0
@@ -738,4 +738,6 @@ xfs_btree_alloc_cursor(int __init xfs_btree_init_cur_caches (void );void xfs_btree_destroy_cur_caches (void ); +int xfs_btree_goto_left_edge (struct xfs_btree_cur *cur); +#endif /* __XFS_BTREE_H__ */
libxfs/xfs_refcount.c +3 -5
@@ -122,11 +122,9 @@ xfs_refcount_btrec_to_irec(/* Simple checks for refcount records. */ xfs_failaddr_t xfs_refcount_check_irec ( -struct xfs_btree_cur *cur, +struct xfs_perag *pag,const struct xfs_refcount_irec *irec) { -struct xfs_perag *pag = cur->bc_ag.pag; -if (irec->rc_blockcount ==0 || irec->rc_blockcount >MAXREFCEXTLEN )return __this_address;
@@ -178,7 +176,7 @@ xfs_refcount_get_rec(return error;xfs_refcount_btrec_to_irec (rec, irec); - fa =xfs_refcount_check_irec (cur, irec); + fa =xfs_refcount_check_irec (cur->bc_ag.pag, irec);if (fa)return xfs_refcount_complain_bad_rec (cur, fa, irec);
@@ -1898,7 +1896,7 @@ xfs_refcount_recover_extent(INIT_LIST_HEAD (&rr->rr_list);xfs_refcount_btrec_to_irec (rec, &rr->rr_rrec); -if (xfs_refcount_check_irec (cur, &rr->rr_rrec) !=NULL || +if (xfs_refcount_check_irec (cur->bc_ag.pag, &rr->rr_rrec) !=NULL ||XFS_IS_CORRUPT (cur->bc_mp, rr->rr_rrec.rc_domain !=XFS_REFC_DOMAIN_COW )) {kfree (rr);
libxfs/xfs_refcount.h +1 -1
@@ -117,7 +117,7 @@ extern int xfs_refcount_has_records(struct xfs_btree_cur *cur,union xfs_btree_rec ;extern void xfs_refcount_btrec_to_irec (const union xfs_btree_rec *rec,struct xfs_refcount_irec *irec); -xfs_failaddr_t xfs_refcount_check_irec (struct xfs_btree_cur *cur, +xfs_failaddr_t xfs_refcount_check_irec (struct xfs_perag *pag,const struct xfs_refcount_irec *irec);extern int xfs_refcount_insert (struct xfs_btree_cur *cur,struct xfs_refcount_irec *irec,int *stat);
libxfs/xfs_refcount_btree.c +12 -1
@@ -225,7 +225,18 @@ xfs_refcountbt_verify( level =be16_to_cpu (block->bb_level);if (pag &&xfs_perag_initialised_agf (pag)) { -if (level >= pag->pagf_refcount_level) +unsigned int maxlevel = pag->pagf_refcount_level; + +#ifdef CONFIG_XFS_ONLINE_REPAIR +/* +* Online repair could be rewriting the refcount btree, so +* we'll validate against the larger of either tree while this +* is going on. +*/ + maxlevel = max_t(unsigned int, maxlevel, + pag->pagf_repair_refcount_level); +#endif +if (level >= maxlevel)return __this_address; }else if (level >= mp->m_refc_maxlevels)return __this_address;
PATCH 37/67 xfs: dont cast to char * for XFS_DFORK_*PTR macros
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 6b5d917780219d0d8f8e2cefefcb6f50987d0fa3
Code in the next patch will assign the return value of XFS_DFORK_*PTR macros to a struct pointer. gcc complains about casting char* strings to struct pointers, so let's fix the macro's cast to void* to shut up the warnings.
While we're at it, fix one of the scrub tests that uses PTR to use BOFF instead for a simpler integer comparison, since other linters whine about char* and void* comparisons.
Can't satisfy all these dman bots.
--- libxfs/xfs_format.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
libxfs/xfs_format.h +1 -1
@@ -1008,7 +1008,7 @@ enum xfs_dinode_fmt {* Return pointers to the data or attribute forks. */ #define XFS_DFORK_DPTR (dip) \ - ((char *)dip + xfs_dinode_size(dip->di_version)) + ((void *)dip + xfs_dinode_size(dip->di_version))#define XFS_DFORK_APTR (dip) \ (XFS_DFORK_DPTR(dip) + XFS_DFORK_BOFF(dip))#define XFS_DFORK_PTR (dip,w) \
PATCH 38/67 xfs: set inode sick state flags when we zap either ondisk fork
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: d9041681dd2f5334529a68868c9266631c384de4
In a few patches, we'll add some online repair code that tries to massage the ondisk inode record just enough to get it to pass the inode verifiers so that we can continue with more file repairs. Part of that massaging can include zapping the ondisk forks to clear errors. After that point, the bmap fork repair functions will rebuild the zapped forks.
Christoph asked for stronger protections against online repair zapping a fork to get the inode to load vs. other threads trying to access the partially repaired file. Do this by adding a special "[DA]FORK_ZAPPED" inode health flag whenever repair zaps a fork, and sprinkling checks for that flag into the various file operations for things that don't like handling an unexpected zero-extents fork.
In practice xfs_scrub will scrub and fix the forks almost immediately after zapping them, so the window is very small. However, if a crash or unmount should occur, we can still detect these zapped inode forks by looking for a zero-extents fork when data was expected.
--- libxfs/xfs_health.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
libxfs/xfs_health.h +10 -0
@@ -68,6 +68,11 @@ struct xfs_fsop_geom;#define XFS_SICK_INO_SYMLINK (1 << 6)/* symbolic link remote target */ #define XFS_SICK_INO_PARENT (1 << 7)/* parent pointers */ +#define XFS_SICK_INO_BMBTD_ZAPPED (1 << 8)/* data fork erased */ +#define XFS_SICK_INO_BMBTA_ZAPPED (1 << 9)/* attr fork erased */ +#define XFS_SICK_INO_DIR_ZAPPED (1 << 10)/* directory erased */ +#define XFS_SICK_INO_SYMLINK_ZAPPED (1 << 11)/* symlink erased */ +/* Primary evidence of health problems in a given group. */ #define XFS_SICK_FS_PRIMARY (XFS_SICK_FS_COUNTERS | \ XFS_SICK_FS_UQUOTA | \
@@ -97,6 +102,11 @@ struct xfs_fsop_geom; XFS_SICK_INO_SYMLINK | \ XFS_SICK_INO_PARENT) +#define XFS_SICK_INO_ZAPPED (XFS_SICK_INO_BMBTD_ZAPPED | \ + XFS_SICK_INO_BMBTA_ZAPPED | \ + XFS_SICK_INO_DIR_ZAPPED | \ + XFS_SICK_INO_SYMLINK_ZAPPED) +/* These functions must be provided by the xfs implementation. */ void xfs_fs_mark_sick (struct xfs_mount *mp,unsigned int mask);
PATCH 39/67 xfs: zap broken inode forks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: e744cef206055954517648070d2b3aaa3d2515ba
Determine if inode fork damage is responsible for the inode being unable to pass the ifork verifiers in xfs_iget and zap the fork contents if this is true. Once this is done the fork will be empty but we'll be able to construct an in-core inode, and a subsequent call to the inode fork repair ioctl will search the rmapbt to rebuild the records that were in the fork.
--- libxfs/xfs_attr_leaf.c | 13 +++---------- libxfs/xfs_attr_leaf.h | 3 ++- libxfs/xfs_bmap.c | 22 ++++++++++++++++------ libxfs/xfs_bmap.h | 2 ++ libxfs/xfs_dir2_priv.h | 3 ++- libxfs/xfs_dir2_sf.c | 13 +++---------- libxfs/xfs_inode_fork.c | 33 ++++++++++++++++++++++++++------- libxfs/xfs_shared.h | 2 +- libxfs/xfs_symlink_remote.c | 8 ++------ 9 files changed, 57 insertions(+), 42 deletions(-)
libxfs/xfs_attr_leaf.c +3 -10
@@ -1037,23 +1037,16 @@ xfs_attr_shortform_allfit(return xfs_attr_shortform_bytesfit (dp, bytes); } -/* Verify the consistency of an inline attribute fork. */ +/* Verify the consistency of a raw inline attribute fork. */ xfs_failaddr_t xfs_attr_shortform_verify ( -struct xfs_inode *ip) +struct xfs_attr_shortform *sfp, +size_t size) { -struct xfs_attr_shortform *sfp;struct xfs_attr_sf_entry *sfep;struct xfs_attr_sf_entry *next_sfep;char *endp; -struct xfs_ifork *ifp;int i; -int64_t size; - -ASSERT (ip->i_af.if_format ==XFS_DINODE_FMT_LOCAL ); - ifp =xfs_ifork_ptr (ip,XFS_ATTR_FORK ); - sfp = (struct xfs_attr_shortform *)ifp->if_u1.if_data; - size = ifp->if_bytes;/* * Give up if the attribute is way too short.
libxfs/xfs_attr_leaf.h +2 -1
@@ -56,7 +56,8 @@ int xfs_attr_sf_findname(struct xfs_da_args *args, unsigned int *basep);int xfs_attr_shortform_allfit (struct xfs_buf *bp,struct xfs_inode *dp);int xfs_attr_shortform_bytesfit (struct xfs_inode *dp,int bytes); -xfs_failaddr_t xfs_attr_shortform_verify (struct xfs_inode *ip); +xfs_failaddr_t xfs_attr_shortform_verify (struct xfs_attr_shortform *sfp, +size_t size);void xfs_attr_fork_remove (struct xfs_inode *ip,struct xfs_trans *tp);/*
libxfs/xfs_bmap.c +16 -6
@@ -6162,19 +6162,18 @@ xfs_bmap_finish_one(return error; } -/* Check that an inode's extent does not have invalid flags or bad ranges. */ +/* Check that an extent does not have invalid flags or bad ranges. */ xfs_failaddr_t -xfs_bmap_validate_extent ( -struct xfs_inode *ip, +xfs_bmap_validate_extent_raw ( +struct xfs_mount *mp, +bool rtfile,int whichfork,struct xfs_bmbt_irec *irec) { -struct xfs_mount *mp = ip->i_mount; -if (!xfs_verify_fileext (mp, irec->br_startoff, irec->br_blockcount))return __this_address; -if (XFS_IS_REALTIME_INODE (ip) && whichfork ==XFS_DATA_FORK ) { +if (rtfile && whichfork ==XFS_DATA_FORK ) {if (!xfs_verify_rtbext (mp, irec->br_startblock, irec->br_blockcount))return __this_address;
@@ -6204,3 +6203,14 @@ xfs_bmap_intent_destroy_cache(void)kmem_cache_destroy (xfs_bmap_intent_cache); xfs_bmap_intent_cache =NULL ; } + +/* Check that an inode's extent does not have invalid flags or bad ranges. */ +xfs_failaddr_t +xfs_bmap_validate_extent ( +struct xfs_inode *ip, +int whichfork, +struct xfs_bmbt_irec *irec) +{ +return xfs_bmap_validate_extent_raw (ip->i_mount, +XFS_IS_REALTIME_INODE (ip), whichfork, irec); +}
libxfs/xfs_bmap.h +2 -0
@@ -263,6 +263,8 @@ static inline uint32_t xfs_bmap_fork_to_state(int whichfork) } } +xfs_failaddr_t xfs_bmap_validate_extent_raw (struct xfs_mount *mp,bool rtfile, +int whichfork,struct xfs_bmbt_irec *irec);xfs_failaddr_t xfs_bmap_validate_extent (struct xfs_inode *ip,int whichfork,struct xfs_bmbt_irec *irec); int xfs_bmap_complain_bad_rec(struct xfs_inode *ip, int whichfork,
libxfs/xfs_dir2_priv.h +2 -1
@@ -175,7 +175,8 @@ extern int xfs_dir2_sf_create(struct xfs_da_args *args, xfs_ino_t pino);extern int xfs_dir2_sf_lookup (struct xfs_da_args *args);extern int xfs_dir2_sf_removename (struct xfs_da_args *args);extern int xfs_dir2_sf_replace (struct xfs_da_args *args); -extern xfs_failaddr_t xfs_dir2_sf_verify (struct xfs_inode *ip); +xfs_failaddr_t xfs_dir2_sf_verify (struct xfs_mount *mp, +struct xfs_dir2_sf_hdr *sfp,int64_t size);int xfs_dir2_sf_entsize (struct xfs_mount *mp,struct xfs_dir2_sf_hdr *hdr,int len); void xfs_dir2_sf_put_ino(struct xfs_mount *mp, struct xfs_dir2_sf_hdr *hdr,
libxfs/xfs_dir2_sf.c +3 -10
@@ -707,11 +707,10 @@ xfs_dir2_sf_check(/* Verify the consistency of an inline directory. */ xfs_failaddr_t xfs_dir2_sf_verify ( -struct xfs_inode *ip) +struct xfs_mount *mp, +struct xfs_dir2_sf_hdr *sfp, +int64_t size) { -struct xfs_mount *mp = ip->i_mount; -struct xfs_ifork *ifp =xfs_ifork_ptr (ip,XFS_DATA_FORK ); -struct xfs_dir2_sf_hdr *sfp;struct xfs_dir2_sf_entry *sfep;struct xfs_dir2_sf_entry *next_sfep;char *endp;
@@ -719,15 +718,9 @@ xfs_dir2_sf_verify(int i;int i8count;int offset; -int64_t size;int error;uint8_t filetype; -ASSERT (ifp->if_format ==XFS_DINODE_FMT_LOCAL ); - - sfp = (struct xfs_dir2_sf_hdr *)ifp->if_u1.if_data; - size = ifp->if_bytes; -/* * Give up if the directory is way too short. */
libxfs/xfs_inode_fork.c +26 -7
@@ -700,12 +700,22 @@ xfs_ifork_verify_local_data(xfs_failaddr_t fa =NULL ;switch (VFS_I (ip)->i_mode &S_IFMT ) { -case S_IFDIR : - fa =xfs_dir2_sf_verify (ip); +case S_IFDIR : { +struct xfs_mount *mp = ip->i_mount; +struct xfs_ifork *ifp =xfs_ifork_ptr (ip,XFS_DATA_FORK ); +struct xfs_dir2_sf_hdr *sfp; + + sfp = (struct xfs_dir2_sf_hdr *)ifp->if_u1.if_data; + fa =xfs_dir2_sf_verify (mp, sfp, ifp->if_bytes);break ; -case S_IFLNK : - fa =xfs_symlink_shortform_verify (ip); + } +case S_IFLNK : { +struct xfs_ifork *ifp =xfs_ifork_ptr (ip,XFS_DATA_FORK ); + + fa =xfs_symlink_shortform_verify (ifp->if_u1.if_data, + ifp->if_bytes);break ; + }default :break ; }
@@ -727,11 +737,20 @@ xfs_ifork_verify_local_attr(struct xfs_ifork *ifp = &ip->i_af;xfs_failaddr_t fa; -if (!xfs_inode_has_attr_fork (ip)) +if (!xfs_inode_has_attr_fork (ip)) { fa = __this_address; -else - fa =xfs_attr_shortform_verify (ip); + }else { +struct xfs_attr_shortform *sfp; +struct xfs_ifork *ifp; +int64_t size; +ASSERT (ip->i_af.if_format ==XFS_DINODE_FMT_LOCAL ); + ifp =xfs_ifork_ptr (ip,XFS_ATTR_FORK ); + sfp = (struct xfs_attr_shortform *)ifp->if_u1.if_data; + size = ifp->if_bytes; + + fa =xfs_attr_shortform_verify (sfp, size); + }if (fa) {xfs_inode_verifier_error (ip, -EFSCORRUPTED ,"attr fork" , ifp->if_u1.if_data, ifp->if_bytes, fa);
libxfs/xfs_shared.h +1 -1
@@ -139,7 +139,7 @@ bool xfs_symlink_hdr_ok(xfs_ino_t ino, uint32_t offset, uint32_t size, struct xfs_buf *bp);void xfs_symlink_local_to_remote (struct xfs_trans *tp,struct xfs_buf *bp,struct xfs_inode *ip,struct xfs_ifork *ifp); -xfs_failaddr_t xfs_symlink_shortform_verify (struct xfs_inode *ip); +xfs_failaddr_t xfs_symlink_shortform_verify (void *sfp,int64_t size);/* Computed inode geometry for the filesystem. */ struct xfs_ino_geometry {
libxfs/xfs_symlink_remote.c +2 -6
@@ -199,15 +199,11 @@ xfs_symlink_local_to_remote(*/ xfs_failaddr_t xfs_symlink_shortform_verify ( -struct xfs_inode *ip) +void *sfp, +int64_t size) { -struct xfs_ifork *ifp =xfs_ifork_ptr (ip,XFS_DATA_FORK ); -char *sfp = (char *)ifp->if_u1.if_data; -int size = ifp->if_bytes;char *endp = sfp + size; -ASSERT (ifp->if_format ==XFS_DINODE_FMT_LOCAL ); -/* * Zero length symlinks should never occur in memory as they are * never allowed to exist on disk.
PATCH 40/67 xfs: repair inode fork block mapping data structures
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 8f71bede8efd820627ac05c19eac2758214bc896
Use the reverse-mapping btree information to rebuild an inode block map. Update the btree bulk loading code as necessary to support inode rooted btrees and fix some bitrot problems.
--- libxfs/xfs_bmap_btree.c | 121 +++++++++++++++++++++++++++++++++++++------- libxfs/xfs_bmap_btree.h | 5 ++ libxfs/xfs_btree_staging.c | 11 +++- libxfs/xfs_btree_staging.h | 2 - libxfs/xfs_iext_tree.c | 23 ++++++-- libxfs/xfs_inode_fork.c | 1 libxfs/xfs_inode_fork.h | 3 + 7 files changed, 136 insertions(+), 30 deletions(-)
libxfs/xfs_bmap_btree.c +101 -20
@@ -15,6 +15,7 @@#include "xfs_trans.h" #include "xfs_alloc.h" #include "xfs_btree.h" +#include "xfs_btree_staging.h" #include "xfs_bmap_btree.h" #include "xfs_bmap.h" #include "xfs_trace.h"
@@ -286,10 +287,7 @@ xfs_bmbt_get_minrecs( int level) {if (level == cur->bc_nlevels -1 ) { -struct xfs_ifork *ifp; - - ifp =xfs_ifork_ptr (cur->bc_ino.ip, - cur->bc_ino.whichfork); +struct xfs_ifork *ifp =xfs_btree_ifork_ptr (cur);return xfs_bmbt_maxrecs (cur->bc_mp, ifp->if_broot_bytes, level ==0 ) /2 ;
@@ -304,10 +302,7 @@ xfs_bmbt_get_maxrecs( int level) {if (level == cur->bc_nlevels -1 ) { -struct xfs_ifork *ifp; - - ifp =xfs_ifork_ptr (cur->bc_ino.ip, - cur->bc_ino.whichfork); +struct xfs_ifork *ifp =xfs_btree_ifork_ptr (cur);return xfs_bmbt_maxrecs (cur->bc_mp, ifp->if_broot_bytes, level ==0 );
@@ -541,23 +536,19 @@ static const struct xfs_btree_ops xfs_bmbt_ops = { .keys_contiguous = xfs_bmbt_keys_contiguous, }; -/* -* Allocate a new bmap btree cursor. -*/ -struct xfs_btree_cur */* new bmap btree cursor */ -xfs_bmbt_init_cursor ( -struct xfs_mount *mp,/* file system mount point */ -struct xfs_trans *tp,/* transaction pointer */ -struct xfs_inode *ip,/* inode owning the btree */ -int whichfork)/* data or attr fork */ +static struct xfs_btree_cur * +xfs_bmbt_init_common ( +struct xfs_mount *mp, +struct xfs_trans *tp, +struct xfs_inode *ip, +int whichfork) { -struct xfs_ifork *ifp =xfs_ifork_ptr (ip, whichfork);struct xfs_btree_cur *cur; +ASSERT (whichfork !=XFS_COW_FORK ); cur =xfs_btree_alloc_cursor (mp, tp,XFS_BTNUM_BMAP , mp->m_bm_maxlevels[whichfork], xfs_bmbt_cur_cache); - cur->bc_nlevels =be16_to_cpu (ifp->if_broot->bb_level) +1 ; cur->bc_statoff =XFS_STATS_CALC_INDEX (xs_bmbt_2); cur->bc_ops = &xfs_bmbt_ops;
@@ -565,10 +556,30 @@ xfs_bmbt_init_cursor(if (xfs_has_crc (mp)) cur->bc_flags |=XFS_BTREE_CRC_BLOCKS ; - cur->bc_ino.forksize =xfs_inode_fork_size (ip, whichfork); cur->bc_ino.ip = ip; cur->bc_ino.allocated =0 ; cur->bc_ino.flags =0 ; + +return cur; +} + +/* +* Allocate a new bmap btree cursor. +*/ +struct xfs_btree_cur * +xfs_bmbt_init_cursor ( +struct xfs_mount *mp, +struct xfs_trans *tp, +struct xfs_inode *ip, +int whichfork) +{ +struct xfs_ifork *ifp =xfs_ifork_ptr (ip, whichfork); +struct xfs_btree_cur *cur; + + cur =xfs_bmbt_init_common (mp, tp, ip, whichfork); + + cur->bc_nlevels =be16_to_cpu (ifp->if_broot->bb_level) +1 ; + cur->bc_ino.forksize =xfs_inode_fork_size (ip, whichfork); cur->bc_ino.whichfork = whichfork;return cur;
@@ -585,6 +596,76 @@ xfs_bmbt_block_maxrecs(return blocklen / (sizeof (xfs_bmbt_key_t) +sizeof (xfs_bmbt_ptr_t)); } +/* +* Allocate a new bmap btree cursor for reloading an inode block mapping data +* structure. Note that callers can use the staged cursor to reload extents +* format inode forks if they rebuild the iext tree and commit the staged +* cursor immediately. +*/ +struct xfs_btree_cur * +xfs_bmbt_stage_cursor ( +struct xfs_mount *mp, +struct xfs_inode *ip, +struct xbtree_ifakeroot *ifake) +{ +struct xfs_btree_cur *cur; +struct xfs_btree_ops *ops; + +/* data fork always has larger maxheight */ + cur =xfs_bmbt_init_common (mp,NULL , ip,XFS_DATA_FORK ); + cur->bc_nlevels = ifake->if_levels; + cur->bc_ino.forksize = ifake->if_fork_size; + +/* Don't let anyone think we're attached to the real fork yet. */ + cur->bc_ino.whichfork =-1 ; +xfs_btree_stage_ifakeroot (cur, ifake, &ops); + ops->update_cursor =NULL ; +return cur; +} + +/* +* Swap in the new inode fork root. Once we pass this point the newly rebuilt +* mappings are in place and we have to kill off any old btree blocks. +*/ +void +xfs_bmbt_commit_staged_btree ( +struct xfs_btree_cur *cur, +struct xfs_trans *tp, +int whichfork) +{ +struct xbtree_ifakeroot *ifake = cur->bc_ino.ifake; +struct xfs_ifork *ifp; +static const short brootflag[2 ] = {XFS_ILOG_DBROOT ,XFS_ILOG_ABROOT }; +static const short extflag[2 ] = {XFS_ILOG_DEXT ,XFS_ILOG_AEXT }; +int flags =XFS_ILOG_CORE ; + +ASSERT (cur->bc_flags &XFS_BTREE_STAGING ); +ASSERT (whichfork !=XFS_COW_FORK ); + +/* +* Free any resources hanging off the real fork, then shallow-copy the +* staging fork's contents into the real fork to transfer everything +* we just built. +*/ + ifp =xfs_ifork_ptr (cur->bc_ino.ip, whichfork); +xfs_idestroy_fork (ifp); +memcpy (ifp, ifake->if_fork,sizeof (struct xfs_ifork )); + +switch (ifp->if_format) { +case XFS_DINODE_FMT_EXTENTS : + flags |= extflag[whichfork]; +break ; +case XFS_DINODE_FMT_BTREE : + flags |= brootflag[whichfork]; +break ; +default : +ASSERT (0 ); +break ; + } +xfs_trans_log_inode (tp, cur->bc_ino.ip, flags); +xfs_btree_commit_ifakeroot (cur, tp, whichfork, &xfs_bmbt_ops); +} +/* * Calculate number of records in a bmap btree block. */
libxfs/xfs_bmap_btree.h +5 -0
@@ -11,6 +11,7 @@ struct xfs_btree_block;struct xfs_mount ;struct xfs_inode ;struct xfs_trans ; +struct xbtree_ifakeroot ;/* * Btree block header size depends on a superblock flag.
@@ -106,6 +107,10 @@ extern int xfs_bmbt_change_owner(struct xfs_trans *tp, struct xfs_inode *ip,extern struct xfs_btree_cur *xfs_bmbt_init_cursor (struct xfs_mount *,struct xfs_trans *,struct xfs_inode *,int ); +struct xfs_btree_cur *xfs_bmbt_stage_cursor (struct xfs_mount *mp, +struct xfs_inode *ip,struct xbtree_ifakeroot *ifake); +void xfs_bmbt_commit_staged_btree (struct xfs_btree_cur *cur, +struct xfs_trans *tp,int whichfork);extern unsigned long long xfs_bmbt_calc_size (struct xfs_mount *mp,unsigned long long len);
libxfs/xfs_btree_staging.c +9 -2
@@ -405,7 +405,7 @@ xfs_btree_bload_prep_block(ASSERT (*bpp ==NULL );/* Allocate a new incore btree root block. */ - new_size = bbl->iroot_size (cur, nr_this_block, priv); + new_size = bbl->iroot_size (cur, level, nr_this_block, priv); ifp->if_broot =kmem_zalloc (new_size,0 ); ifp->if_broot_bytes = (int )new_size;
@@ -596,7 +596,14 @@ xfs_btree_bload_level_geometry(unsigned int desired_npb;unsigned int maxnr; - maxnr = cur->bc_ops->get_maxrecs (cur, level); +/* +* Compute the absolute maximum number of records that we can store in +* the ondisk block or inode root. +*/ +if (cur->bc_ops->get_dmaxrecs) + maxnr = cur->bc_ops->get_dmaxrecs (cur, level); +else + maxnr = cur->bc_ops->get_maxrecs (cur, level);/* * Compute the number of blocks we need to fill each block with the
libxfs/xfs_btree_staging.h +1 -1
@@ -53,7 +53,7 @@ typedef int (*xfs_btree_bload_get_records_fn)(struct xfs_btree_cur *cur,typedef int (*xfs_btree_bload_claim_block_fn )(struct xfs_btree_cur *cur,union xfs_btree_ptr *ptr,void *priv);typedef size_t (*xfs_btree_bload_iroot_size_fn )(struct xfs_btree_cur *cur, -unsigned int nr_this_level,void *priv); +unsigned int level,unsigned int nr_this_level,void *priv);struct xfs_btree_bload {/*
libxfs/xfs_iext_tree.c +16 -7
@@ -622,13 +622,11 @@ static inline void xfs_iext_inc_seq(struct xfs_ifork *ifp) }void -xfs_iext_insert ( -struct xfs_inode *ip, +xfs_iext_insert_raw ( +struct xfs_ifork *ifp,struct xfs_iext_cursor *cur, -struct xfs_bmbt_irec *irec, -int state) +struct xfs_bmbt_irec *irec) { -struct xfs_ifork *ifp =xfs_iext_state_to_fork (ip, state);xfs_fileoff_t offset = irec->br_startoff;struct xfs_iext_leaf *new =NULL ;int nr_entries, i;
@@ -662,12 +660,23 @@ xfs_iext_insert(xfs_iext_set (cur_rec (cur), irec); ifp->if_bytes +=sizeof (struct xfs_iext_rec ); -trace_xfs_iext_insert (ip, cur, state, _RET_IP_); -if (new)xfs_iext_insert_node (ifp,xfs_iext_leaf_key (new,0 ), new,2 ); } +void +xfs_iext_insert ( +struct xfs_inode *ip, +struct xfs_iext_cursor *cur, +struct xfs_bmbt_irec *irec, +int state) +{ +struct xfs_ifork *ifp =xfs_iext_state_to_fork (ip, state); + +xfs_iext_insert_raw (ifp, cur, irec); +trace_xfs_iext_insert (ip, cur, state, _RET_IP_); +} +static struct xfs_iext_node * xfs_iext_rebalance_node( struct xfs_iext_node *parent,
libxfs/xfs_inode_fork.c +1 -0
@@ -518,6 +518,7 @@ xfs_idata_realloc( ifp->if_bytes = new_size; } +/* Free all memory and reset a fork back to its initial state. */ void xfs_idestroy_fork (struct xfs_ifork *ifp)
libxfs/xfs_inode_fork.h +3 -0
@@ -180,6 +180,9 @@ void xfs_init_local_fork(struct xfs_inode *ip, int whichfork, const void *data, int64_t size);xfs_extnum_t xfs_iext_count (struct xfs_ifork *ifp); +void xfs_iext_insert_raw (struct xfs_ifork *ifp, +struct xfs_iext_cursor *cur, +struct xfs_bmbt_irec *irec);void xfs_iext_insert (struct xfs_inode *,struct xfs_iext_cursor *cur,struct xfs_bmbt_irec *,int ); void xfs_iext_remove(struct xfs_inode *, struct xfs_iext_cursor *,
PATCH 41/67 xfs: create a ranged query function for refcount btrees
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: d12bf8bac87a0d93e6e5fab67f399d1e3d3d5767
Implement ranged queries for refcount records. The next patch will use this to scan refcount data.
--- libxfs/xfs_refcount.c | 41 +++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_refcount.h | 10 ++++++++++ 2 files changed, 51 insertions(+)
libxfs/xfs_refcount.c +41 -0
@@ -2030,6 +2030,47 @@ xfs_refcount_has_records(return xfs_btree_has_records (cur, &low, &high,NULL , outcome); } +struct xfs_refcount_query_range_info { +xfs_refcount_query_range_fn fn; +void *priv; +}; + +/* Format btree record and pass to our callback. */ +STATIC int +xfs_refcount_query_range_helper ( +struct xfs_btree_cur *cur, +const union xfs_btree_rec *rec, +void *priv) +{ +struct xfs_refcount_query_range_info *query = priv; +struct xfs_refcount_irec irec; +xfs_failaddr_t fa; + +xfs_refcount_btrec_to_irec (rec, &irec); + fa =xfs_refcount_check_irec (cur->bc_ag.pag, &irec); +if (fa) +return xfs_refcount_complain_bad_rec (cur, fa, &irec); + +return query->fn (cur, &irec, query->priv); +} + +/* Find all refcount records between two keys. */ +int +xfs_refcount_query_range ( +struct xfs_btree_cur *cur, +const struct xfs_refcount_irec *low_rec, +const struct xfs_refcount_irec *high_rec, +xfs_refcount_query_range_fn fn, +void *priv) +{ +union xfs_btree_irec low_brec = { .rc = *low_rec }; +union xfs_btree_irec high_brec = { .rc = *high_rec }; +struct xfs_refcount_query_range_info query = { .priv = priv, .fn = fn }; + +return xfs_btree_query_range (cur, &low_brec, &high_brec, + xfs_refcount_query_range_helper, &query); +} +int __init xfs_refcount_intent_init_cache (void ) {
libxfs/xfs_refcount.h +10 -0
@@ -127,4 +127,14 @@ extern struct kmem_cache *xfs_refcount_intent_cache;int __init xfs_refcount_intent_init_cache (void );void xfs_refcount_intent_destroy_cache (void ); +typedef int (*xfs_refcount_query_range_fn )( +struct xfs_btree_cur *cur, +const struct xfs_refcount_irec *rec, +void *priv); + +int xfs_refcount_query_range (struct xfs_btree_cur *cur, +const struct xfs_refcount_irec *low_rec, +const struct xfs_refcount_irec *high_rec, +xfs_refcount_query_range_fn fn,void *priv); +#endif /* __XFS_REFCOUNT_H__ */
PATCH 42/67 xfs: create a new inode fork block unmap helper
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: a59eb5fc21b2a6dc160ee6cdf77f20bc186a88fd
Create a new helper to unmap blocks from an inode's fork.
--- libxfs/xfs_bmap.c | 41 ++++++++++++++++++++++++++++++++++++++++- libxfs/xfs_bmap.h | 5 ++--- 2 files changed, 42 insertions(+), 4 deletions(-)
libxfs/xfs_bmap.c +40 -1
@@ -5233,7 +5233,7 @@ xfs_bmap_del_extent_real(* that value. If not all extents in the block range can be removed then * *done is set. */ -int /* error */ +static int __xfs_bunmapi( struct xfs_trans *tp, /* transaction pointer */ struct xfs_inode *ip, /* incore inode */
@@ -6214,3 +6214,42 @@ xfs_bmap_validate_extent(return xfs_bmap_validate_extent_raw (ip->i_mount,XFS_IS_REALTIME_INODE (ip), whichfork, irec); } + +/* +* Used in xfs_itruncate_extents(). This is the maximum number of extents +* freed from a file in a single transaction. +*/ +#define XFS_ITRUNC_MAX_EXTENTS 2 + +/* +* Unmap every extent in part of an inode's fork. We don't do any higher level +* invalidation work at all. +*/ +int +xfs_bunmapi_range ( +struct xfs_trans **tpp, +struct xfs_inode *ip, +uint32_t flags, +xfs_fileoff_t startoff, +xfs_fileoff_t endoff) +{ +xfs_filblks_t unmap_len = endoff - startoff +1 ; +int error =0 ; + +ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL )); + +while (unmap_len >0 ) { +ASSERT ((*tpp)->t_highest_agno ==NULLAGNUMBER ); + error =__xfs_bunmapi (*tpp, ip, startoff, &unmap_len, flags, +XFS_ITRUNC_MAX_EXTENTS ); +if (error) + goto out; + +/* free the just unmapped extents */ + error =xfs_defer_finish (tpp); +if (error) + goto out; + } +out: +return error; +}
libxfs/xfs_bmap.h +2 -3
@@ -190,9 +190,6 @@ int xfs_bmapi_read(struct xfs_inode *ip, xfs_fileoff_t bno,int xfs_bmapi_write (struct xfs_trans *tp,struct xfs_inode *ip,xfs_fileoff_t bno,xfs_filblks_t len,uint32_t flags,xfs_extlen_t total,struct xfs_bmbt_irec *mval,int *nmap); -int __xfs_bunmapi (struct xfs_trans *tp,struct xfs_inode *ip, -xfs_fileoff_t bno,xfs_filblks_t *rlen,uint32_t flags, -xfs_extnum_t nexts);int xfs_bunmapi (struct xfs_trans *tp,struct xfs_inode *ip,xfs_fileoff_t bno,xfs_filblks_t len,uint32_t flags,xfs_extnum_t nexts,int *done);
@@ -273,6 +270,8 @@ int xfs_bmap_complain_bad_rec(struct xfs_inode *ip, int whichfork,int xfs_bmapi_remap (struct xfs_trans *tp,struct xfs_inode *ip,xfs_fileoff_t bno,xfs_filblks_t len,xfs_fsblock_t startblock,uint32_t flags); +int xfs_bunmapi_range (struct xfs_trans **tpp,struct xfs_inode *ip, +uint32_t flags,xfs_fileoff_t startoff,xfs_fileoff_t endoff);extern struct kmem_cache *xfs_bmap_intent_cache;
PATCH 43/67 xfs: improve dquot iteration for scrub
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 21d7500929c8a0b10e22a6755850c6f9a9280284
Upon a closer inspection of the quota record scrubber, I noticed that dqiterate wasn't actually walking all possible dquots for the mapped blocks in the quota file. This is due to xfs_qm_dqget_next skipping all XFS_IS_DQUOT_UNINITIALIZED dquots.
For a fsck program, we really want to look at all the dquots, even if all counters and limits in the dquot record are zero. Rewrite the implementation to do this, as well as switching to an iterator paradigm to reduce the number of indirect calls.
This enables removal of the old broken dqiterate code from xfs_dquot.c.
--- libxfs/xfs_format.h | 3 +++ 1 file changed, 3 insertions(+)
libxfs/xfs_format.h +3 -0
@@ -1272,6 +1272,9 @@ static inline time64_t xfs_dq_bigtime_to_unix(uint32_t ondisk_seconds)#define XFS_DQ_GRACE_MIN ((int64_t)0)#define XFS_DQ_GRACE_MAX ((int64_t)U32_MAX) +/* Maximum id value for a quota record */ +#define XFS_DQ_ID_MAX (U32_MAX) +/* * This is the main portion of the on-disk representation of quota information * for a user. We pad this with some more expansion room to construct the on
PATCH 44/67 xfs: add lock protection when remove perag from radix tree
In reply to Darrick J. WongFrom: Long Li <[email protected]>
Source kernel commit: 07afd3173d0c6d24a47441839a835955ec6cf0d4
Take mp->m_perag_lock for deletions from the perag radix tree in xfs_initialize_perag to prevent racing with tagging operations. Lookups are fine - they are RCU protected so already deal with the tree changing shape underneath the lookup - but tagging operations require the tree to be stable while the tags are propagated back up to the root.
Right now there's nothing stopping radix tree tagging from operating while a growfs operation is progress and adding/removing new entries into the radix tree.
Hence we can have traversals that require a stable tree occurring at the same time we are removing unused entries from the radix tree which causes the shape of the tree to change.
Likely this hasn't caused a problem in the past because we are only doing append addition and removal so the active AG part of the tree is not changing shape, but that doesn't mean it is safe. Just making the radix tree modifications serialise against each other is obviously correct.
--- libxfs/xfs_ag.c | 4 ++++ 1 file changed, 4 insertions(+)
libxfs/xfs_ag.c +4 -0
@@ -422,13 +422,17 @@ xfs_initialize_perag( out_remove_pag:xfs_defer_drain_free (&pag->pag_intents_drain); +spin_lock (&mp->m_perag_lock);radix_tree_delete (&mp->m_perag_tree, index); +spin_unlock (&mp->m_perag_lock); out_free_pag:kmem_free (pag); out_unwind_new_pags:/* unwind any prior newly initialized pags */ for (index = first_initialised; index < agcount; index++) { +spin_lock (&mp->m_perag_lock); pag =radix_tree_delete (&mp->m_perag_tree, index); +spin_unlock (&mp->m_perag_lock);if (!pag)break ;xfs_buf_hash_destroy (pag);
PATCH 45/67 xfs: fix perag leak when growfs fails
In reply to Darrick J. WongFrom: Long Li <[email protected]>
Source kernel commit: 7823921887750b39d02e6b44faafdd1cc617c651
During growfs, if new ag in memory has been initialized, however sb_agcount has not been updated, if an error occurs at this time it will cause perag leaks as follows, these new AGs will not been freed during umount , because of these new AGs are not visible(that is included in mp->m_sb.sb_agcount).
unreferenced object 0xffff88810be40200 (size 512):
comm "xfs_growfs", pid 857, jiffies 4294909093
hex dump (first 32 bytes):
00 c0 c1 05 81 88 ff ff 04 00 00 00 00 00 00 00 ................ 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace (crc 381741e2):
[<ffffffff8191aef6>] __kmalloc+0x386/0x4f0 [<ffffffff82553e65>] kmem_alloc+0xb5/0x2f0 [<ffffffff8238dac5>] xfs_initialize_perag+0xc5/0x810 [<ffffffff824f679c>] xfs_growfs_data+0x9bc/0xbc0 [<ffffffff8250b90e>] xfs_file_ioctl+0x5fe/0x14d0 [<ffffffff81aa5194>] __x64_sys_ioctl+0x144/0x1c0 [<ffffffff83c3d81f>] do_syscall_64+0x3f/0xe0 [<ffffffff83e00087>] entry_SYSCALL_64_after_hwframe+0x62/0x6a
unreferenced object 0xffff88810be40800 (size 512):
comm "xfs_growfs", pid 857, jiffies 4294909093
hex dump (first 32 bytes):
20 00 00 00 00 00 00 00 57 ef be dc 00 00 00 00 .......W.......
10 08 e4 0b 81 88 ff ff 10 08 e4 0b 81 88 ff ff ................ backtrace (crc bde50e2d):
[<ffffffff8191b43a>] __kmalloc_node+0x3da/0x540 [<ffffffff81814489>] kvmalloc_node+0x99/0x160 [<ffffffff8286acff>] bucket_table_alloc.isra.0+0x5f/0x400 [<ffffffff8286bdc5>] rhashtable_init+0x405/0x760 [<ffffffff8238dda3>] xfs_initialize_perag+0x3a3/0x810 [<ffffffff824f679c>] xfs_growfs_data+0x9bc/0xbc0 [<ffffffff8250b90e>] xfs_file_ioctl+0x5fe/0x14d0 [<ffffffff81aa5194>] __x64_sys_ioctl+0x144/0x1c0 [<ffffffff83c3d81f>] do_syscall_64+0x3f/0xe0 [<ffffffff83e00087>] entry_SYSCALL_64_after_hwframe+0x62/0x6a
Factor out xfs_free_unused_perag_range() from xfs_initialize_perag(), used for freeing unused perag within a specified range in error handling, included in the error path of the growfs failure.
--- libxfs/xfs_ag.c | 36 ++++++++++++++++++++++++++---------- libxfs/xfs_ag.h | 2 ++ 2 files changed, 28 insertions(+), 10 deletions(-)
libxfs/xfs_ag.c +26 -10
@@ -330,6 +330,31 @@ xfs_agino_range(return __xfs_agino_range (mp,xfs_ag_block_count (mp, agno), first, last); } +/* +* Free perag within the specified AG range, it is only used to free unused +* perags under the error handling path. +*/ +void +xfs_free_unused_perag_range ( +struct xfs_mount *mp, +xfs_agnumber_t agstart, +xfs_agnumber_t agend) +{ +struct xfs_perag *pag; +xfs_agnumber_t index; + +for (index = agstart; index < agend; index++) { +spin_lock (&mp->m_perag_lock); + pag =radix_tree_delete (&mp->m_perag_tree, index); +spin_unlock (&mp->m_perag_lock); +if (!pag) +break ; +xfs_buf_hash_destroy (pag); +xfs_defer_drain_free (&pag->pag_intents_drain); +kmem_free (pag); + } +} + int xfs_initialize_perag( struct xfs_mount *mp,
@@ -429,16 +454,7 @@ xfs_initialize_perag(kmem_free (pag); out_unwind_new_pags:/* unwind any prior newly initialized pags */ -for (index = first_initialised; index < agcount; index++) { -spin_lock (&mp->m_perag_lock); - pag =radix_tree_delete (&mp->m_perag_tree, index); -spin_unlock (&mp->m_perag_lock); -if (!pag) -break ; -xfs_buf_hash_destroy (pag); -xfs_defer_drain_free (&pag->pag_intents_drain); -kmem_free (pag); - } +xfs_free_unused_perag_range (mp, first_initialised, agcount);return error; }
libxfs/xfs_ag.h +2 -0
@@ -143,6 +143,8 @@ __XFS_AG_OPSTATE(prefers_metadata, PREFERS_METADATA) __XFS_AG_OPSTATE(allows_inodes, ALLOWS_INODES) __XFS_AG_OPSTATE(agfl_needs_reset, AGFL_NEEDS_RESET) +void xfs_free_unused_perag_range (struct xfs_mount *mp,xfs_agnumber_t agstart, +xfs_agnumber_t agend);int xfs_initialize_perag (struct xfs_mount *mp,xfs_agnumber_t agcount,xfs_rfsblock_t dcount,xfs_agnumber_t *maxagi);int xfs_initialize_perag_data (struct xfs_mount *mp,xfs_agnumber_t agno);
PATCH 46/67 xfs: remove the xfs_alloc_arg argument to xfs_bmap_btalloc_accounting
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: eef519d746bbfb90cbad4077c2d39d7a359c3282
xfs_bmap_btalloc_accounting only uses the len field from args, but that has just been propagated to ap->length field by the caller.
--- libxfs/xfs_bmap.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-)
libxfs/xfs_bmap.c +9 -10
@@ -3259,8 +3259,7 @@ xfs_bmap_btalloc_select_lengths(/* Update all inode and quota accounting for the allocation we just did. */ static void xfs_bmap_btalloc_accounting ( -struct xfs_bmalloca *ap, -struct xfs_alloc_arg *args) +struct xfs_bmalloca *ap) {if (ap->flags &XFS_BMAPI_COWFORK ) {/*
@@ -3273,7 +3272,7 @@ xfs_bmap_btalloc_accounting(* yet. */ if (ap->wasdel) { -xfs_mod_delalloc (ap->ip->i_mount, -(int64_t )args->len); +xfs_mod_delalloc (ap->ip->i_mount, -(int64_t )ap->length);return ; }
@@ -3285,22 +3284,22 @@ xfs_bmap_btalloc_accounting(* This essentially transfers the transaction quota reservation * to that of a delalloc extent. */ - ap->ip->i_delayed_blks += args->len; + ap->ip->i_delayed_blks += ap->length;xfs_trans_mod_dquot_byino (ap->tp, ap->ip,XFS_TRANS_DQ_RES_BLKS , - -(long )args->len); + -(long )ap->length);return ; }/* data/attr fork only */ - ap->ip->i_nblocks += args->len; + ap->ip->i_nblocks += ap->length;xfs_trans_log_inode (ap->tp, ap->ip,XFS_ILOG_CORE );if (ap->wasdel) { - ap->ip->i_delayed_blks -= args->len; -xfs_mod_delalloc (ap->ip->i_mount, -(int64_t )args->len); + ap->ip->i_delayed_blks -= ap->length; +xfs_mod_delalloc (ap->ip->i_mount, -(int64_t )ap->length); }xfs_trans_mod_dquot_byino (ap->tp, ap->ip, ap->wasdel ?XFS_TRANS_DQ_DELBCOUNT :XFS_TRANS_DQ_BCOUNT , - args->len); + ap->length); } static int
@@ -3374,7 +3373,7 @@ xfs_bmap_process_allocated_extent( ap->offset = orig_offset; else if (ap->offset + ap->length < orig_offset + orig_length) ap->offset = orig_offset + orig_length - ap->length; -xfs_bmap_btalloc_accounting (ap, args); +xfs_bmap_btalloc_accounting (ap); }#ifdef DEBUG
PATCH 47/67 xfs: also use xfs_bmap_btalloc_accounting for RT allocations
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 58643460546da1dc61593fc6fd78762798b4534f
Make xfs_bmap_btalloc_accounting more generic by handling the RT quota reservations and then also use it from xfs_bmap_rtalloc instead of open coding the accounting logic there.
--- libxfs/libxfs_priv.h | 5 ++++- libxfs/xfs_bmap.c | 21 ++++++++++++++------- libxfs/xfs_bmap.h | 2 ++ libxfs/xfs_bmap_btree.c | 1 + 4 files changed, 21 insertions(+), 8 deletions(-)
libxfs/libxfs_priv.h +4 -1
@@ -434,7 +434,10 @@ void __xfs_buf_mark_corrupt(struct xfs_buf *bp, xfs_failaddr_t fa);#define xfs_filestream_select_ag (...) (-ENOSYS)/* quota bits */ -#define xfs_trans_mod_dquot_byino (t,i,f,d) ((void) 0) +#define xfs_trans_mod_dquot_byino(t,i,f,d) ({ \ + uint _f = (f); \ + _f = _f; /* shut up gcc */ \ +})#define xfs_trans_reserve_quota_nblks (t,i,b,n,f) (0)/* hack too silence gcc */
libxfs/xfs_bmap.c +14 -7
@@ -3257,10 +3257,14 @@ xfs_bmap_btalloc_select_lengths( }/* Update all inode and quota accounting for the allocation we just did. */ -static void -xfs_bmap_btalloc_accounting ( +void +xfs_bmap_alloc_account (struct xfs_bmalloca *ap) { +bool isrt =XFS_IS_REALTIME_INODE (ap->ip) && + (ap->flags &XFS_BMAPI_ATTRFORK ); +uint fld; +if (ap->flags &XFS_BMAPI_COWFORK ) {/* * COW fork blocks are in-core only and thus are treated as
@@ -3285,7 +3289,8 @@ xfs_bmap_btalloc_accounting(* to that of a delalloc extent. */ ap->ip->i_delayed_blks += ap->length; -xfs_trans_mod_dquot_byino (ap->tp, ap->ip,XFS_TRANS_DQ_RES_BLKS , +xfs_trans_mod_dquot_byino (ap->tp, ap->ip, isrt ? +XFS_TRANS_DQ_RES_RTBLKS :XFS_TRANS_DQ_RES_BLKS , -(long )ap->length);return ; }
@@ -3296,10 +3301,12 @@ xfs_bmap_btalloc_accounting(if (ap->wasdel) { ap->ip->i_delayed_blks -= ap->length;xfs_mod_delalloc (ap->ip->i_mount, -(int64_t )ap->length); + fld = isrt ?XFS_TRANS_DQ_DELRTBCOUNT :XFS_TRANS_DQ_DELBCOUNT ; + }else { + fld = isrt ?XFS_TRANS_DQ_RTBCOUNT :XFS_TRANS_DQ_BCOUNT ; } -xfs_trans_mod_dquot_byino (ap->tp, ap->ip, - ap->wasdel ?XFS_TRANS_DQ_DELBCOUNT :XFS_TRANS_DQ_BCOUNT , - ap->length); + +xfs_trans_mod_dquot_byino (ap->tp, ap->ip, fld, ap->length); } static int
@@ -3373,7 +3380,7 @@ xfs_bmap_process_allocated_extent( ap->offset = orig_offset; else if (ap->offset + ap->length < orig_offset + orig_length) ap->offset = orig_offset + orig_length - ap->length; -xfs_bmap_btalloc_accounting (ap); +xfs_bmap_alloc_account (ap); }#ifdef DEBUG
libxfs/xfs_bmap.h +2 -0
@@ -116,6 +116,8 @@ static inline int xfs_bmapi_whichfork(uint32_t bmapi_flags)return XFS_DATA_FORK ; } +void xfs_bmap_alloc_account (struct xfs_bmalloca *ap); +/* * Special values for xfs_bmbt_irec_t br_startblock field. */
libxfs/xfs_bmap_btree.c +1 -0
@@ -21,6 +21,7 @@#include "xfs_trace.h" #include "xfs_rmap.h" #include "xfs_ag.h" +#include "xfs_quota_defs.h" static struct kmem_cache *xfs_bmbt_cur_cache;
PATCH 48/67 xfs: return -ENOSPC from xfs_rtallocate_*
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: ce42b5d37527b282d38413c1b5f7283253f6562d
Just return -ENOSPC instead of returning 0 and setting the return rt extent number to NULLRTEXTNO. This is turn removes all users of NULLRTEXTNO, so remove that as well.
--- libxfs/xfs_types.h | 1 - 1 file changed, 1 deletion(-)
libxfs/xfs_types.h +0 -1
@@ -51,7 +51,6 @@ typedef void * xfs_failaddr_t;#define NULLRFSBLOCK ((xfs_rfsblock_t)-1)#define NULLRTBLOCK ((xfs_rtblock_t)-1)#define NULLFILEOFF ((xfs_fileoff_t)-1) -#define NULLRTEXTNO ((xfs_rtxnum_t)-1)#define NULLAGBLOCK ((xfs_agblock_t)-1)#define NULLAGNUMBER ((xfs_agnumber_t)-1)
PATCH 49/67 xfs: indicate if xfs_bmap_adjacent changed ap->blkno
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 676544c27e710aee7f8357f57abd348d98b1ccd4
Add a return value to xfs_bmap_adjacent to indicate if it did change ap->blkno or not.
--- libxfs/libxfs_priv.h | 2 +- libxfs/xfs_bmap.c | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 6 deletions(-)
libxfs/libxfs_priv.h +1 -1
@@ -565,7 +565,7 @@ int xfs_bmap_extsize_align(struct xfs_mount *mp, struct xfs_bmbt_irec *gotp, struct xfs_bmbt_irec *prevp, xfs_extlen_t extsz, int rt, int eof, int delay, int convert, xfs_fileoff_t *offp, xfs_extlen_t *lenp); -void xfs_bmap_adjacent (struct xfs_bmalloca *ap); +bool xfs_bmap_adjacent (struct xfs_bmalloca *ap);int xfs_bmap_last_extent (struct xfs_trans *tp,struct xfs_inode *ip,int whichfork,struct xfs_bmbt_irec *rec,int *is_empty);
libxfs/xfs_bmap.c +14 -5
@@ -3038,7 +3038,8 @@ xfs_bmap_extsize_align(#define XFS_ALLOC_GAP_UNITS 4 -void +/* returns true if ap->blkno was modified */ +bool xfs_bmap_adjacent (struct xfs_bmalloca *ap)/* bmap alloc argument struct */ {
@@ -3073,13 +3074,14 @@ xfs_bmap_adjacent(if (adjust &&ISVALID (ap->blkno + adjust, ap->prev.br_startblock)) ap->blkno += adjust; +return true; }/* * If not at eof, then compare the two neighbor blocks. * Figure out whether either one gives us a good starting point, * and pick the better one. */ -else if (!ap->eof) { +if (!ap->eof) {xfs_fsblock_t gotbno;/* right side block number */ xfs_fsblock_t gotdiff=0 ;/* right side difference */ xfs_fsblock_t prevbno;/* left side block number */
@@ -3159,14 +3161,21 @@ xfs_bmap_adjacent(* If both valid, pick the better one, else the only good * one, else ap->blkno is already set (to 0 or the inode block). */ -if (prevbno !=NULLFSBLOCK && gotbno !=NULLFSBLOCK ) +if (prevbno !=NULLFSBLOCK && gotbno !=NULLFSBLOCK ) { ap->blkno = prevdiff <= gotdiff ? prevbno : gotbno; -else if (prevbno !=NULLFSBLOCK ) +return true; + } +if (prevbno !=NULLFSBLOCK ) { ap->blkno = prevbno; -else if (gotbno !=NULLFSBLOCK ) +return true; + } +if (gotbno !=NULLFSBLOCK ) { ap->blkno = gotbno; +return true; + } }#undef ISVALID +return false; }int
PATCH 50/67 xfs: move xfs_rtget_summary to xfs_rtbitmap.c
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: c2adcfa31ff606264fab6e69129d6d45c9ddb7cb
xfs_rtmodify_summary_int is only used inside xfs_rtbitmap.c and to implement xfs_rtget_summary. Move xfs_rtget_summary to xfs_rtbitmap.c as the exported API and mark xfs_rtmodify_summary_int static.
--- libxfs/xfs_rtbitmap.c | 14 ++++++++++++++ libxfs/xfs_rtbitmap.h | 4 ++-- 2 files changed, 16 insertions(+), 2 deletions(-)
libxfs/xfs_rtbitmap.c +14 -0
@@ -517,6 +517,20 @@ xfs_rtmodify_summary(return xfs_rtmodify_summary_int (args, log, bbno, delta,NULL ); } +/* +* Read and return the summary information for a given extent size, bitmap block +* combination. +*/ +int +xfs_rtget_summary ( +struct xfs_rtalloc_args *args, +int log,/* log2 of extent size */ +xfs_fileoff_t bbno,/* bitmap block number */ +xfs_suminfo_t *sum)/* out: summary info for this block */ +{ +return xfs_rtmodify_summary_int (args, log, bbno,0 , sum); +} +/* Log rtbitmap block from the word @from to the byte before @next. */ static inline void xfs_trans_log_rtbitmap(
libxfs/xfs_rtbitmap.h +2 -2
@@ -321,8 +321,8 @@ int xfs_rtfind_forw(struct xfs_rtalloc_args *args, xfs_rtxnum_t start, xfs_rtxnum_t limit, xfs_rtxnum_t *rtblock);int xfs_rtmodify_range (struct xfs_rtalloc_args *args,xfs_rtxnum_t start,xfs_rtxlen_t len,int val); -int xfs_rtmodify_summary_int (struct xfs_rtalloc_args *args,int log, -xfs_fileoff_t bbno,int delta,xfs_suminfo_t *sum); +int xfs_rtget_summary (struct xfs_rtalloc_args *args,int log, +xfs_fileoff_t bbno,xfs_suminfo_t *sum);int xfs_rtmodify_summary (struct xfs_rtalloc_args *args,int log,xfs_fileoff_t bbno,int delta); int xfs_rtfree_range(struct xfs_rtalloc_args *args, xfs_rtxnum_t start,
PATCH 51/67 xfs: split xfs_rtmodify_summary_int
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: b271b314119eca1fb98a2c4e15304ce562802f0c
Inline the logic of xfs_rtmodify_summary_int into xfs_rtmodify_summary and xfs_rtget_summary instead of having a somewhat awkward helper to share a little bit of code.
--- libxfs/xfs_rtbitmap.c | 90 +++++++++++++++++-------------------------------- 1 file changed, 32 insertions(+), 58 deletions(-)
libxfs/xfs_rtbitmap.c +32 -58
@@ -450,63 +450,9 @@ xfs_trans_log_rtsummary( }/* -* Read and/or modify the summary information for a given extent size, -* bitmap block combination. -* Keeps track of a current summary block, so we don't keep reading -* it from the buffer cache. -* -* Summary information is returned in *sum if specified. -* If no delta is specified, returns summary only. +* Modify the summary information for a given extent size, bitmap block +* combination. */ -int -xfs_rtmodify_summary_int ( -struct xfs_rtalloc_args *args, -int log,/* log2 of extent size */ -xfs_fileoff_t bbno,/* bitmap block number */ -int delta,/* change to make to summary info */ -xfs_suminfo_t *sum)/* out: summary info for this block */ -{ -struct xfs_mount *mp = args->mp; -int error; -xfs_fileoff_t sb;/* summary fsblock */ -xfs_rtsumoff_t so;/* index into the summary file */ -unsigned int infoword; - -/* -* Compute entry number in the summary file. -*/ - so =xfs_rtsumoffs (mp, log, bbno); -/* -* Compute the block number in the summary file. -*/ - sb =xfs_rtsumoffs_to_block (mp, so); - - error =xfs_rtsummary_read_buf (args, sb); -if (error) -return error; - -/* -* Point to the summary information, modify/log it, and/or copy it out. -*/ - infoword =xfs_rtsumoffs_to_infoword (mp, so); -if (delta) { -xfs_suminfo_t val =xfs_suminfo_add (args, infoword, delta); - -if (mp->m_rsum_cache) { -if (val ==0 && log +1 == mp->m_rsum_cache[bbno]) - mp->m_rsum_cache[bbno] = log; -if (val !=0 && log >= mp->m_rsum_cache[bbno]) - mp->m_rsum_cache[bbno] = log +1 ; - } -xfs_trans_log_rtsummary (args, infoword); -if (sum) - *sum = val; - }else if (sum) { - *sum =xfs_suminfo_get (args, infoword); - } -return 0 ; -} - int xfs_rtmodify_summary( struct xfs_rtalloc_args *args,
@@ -514,7 +460,28 @@ xfs_rtmodify_summary( xfs_fileoff_t bbno, /* bitmap block number */ int delta) /* in/out: summary block number */ { - return xfs_rtmodify_summary_int(args, log, bbno, delta, NULL); +struct xfs_mount *mp = args->mp; +xfs_rtsumoff_t so =xfs_rtsumoffs (mp, log, bbno); +unsigned int infoword; +xfs_suminfo_t val; +int error; + + error =xfs_rtsummary_read_buf (args,xfs_rtsumoffs_to_block (mp, so)); +if (error) +return error; + + infoword =xfs_rtsumoffs_to_infoword (mp, so); + val =xfs_suminfo_add (args, infoword, delta); + +if (mp->m_rsum_cache) { +if (val ==0 && log +1 == mp->m_rsum_cache[bbno]) + mp->m_rsum_cache[bbno] = log; +if (val !=0 && log >= mp->m_rsum_cache[bbno]) + mp->m_rsum_cache[bbno] = log +1 ; + } + +xfs_trans_log_rtsummary (args, infoword); +return 0 ; }/*
@@ -528,7 +495,14 @@ xfs_rtget_summary(xfs_fileoff_t bbno,/* bitmap block number */ xfs_suminfo_t *sum) /* out: summary info for this block */ { - return xfs_rtmodify_summary_int(args, log, bbno, 0, sum); + struct xfs_mount *mp = args->mp; +xfs_rtsumoff_t so =xfs_rtsumoffs (mp, log, bbno); +int error; + + error =xfs_rtsummary_read_buf (args,xfs_rtsumoffs_to_block (mp, so)); +if (!error) + *sum =xfs_suminfo_get (args,xfs_rtsumoffs_to_infoword (mp, so)); +return error; }/* Log rtbitmap block from the word @from to the byte before @next. */
PATCH 52/67 xfs: remove rt-wrappers from xfs_format.h
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 3abfe6c2759e2e3000b13f8ce8a1a325e80987a1
xfs_format.h has a bunch odd wrappers for helper functions and mount structure access using RT* prefixes. Replace them with their open coded versions (for those that weren't entirely unused) and remove the wrappers.
--- db/check.c | 4 ++-- libxfs/xfs_format.h | 8 -------- libxfs/xfs_rtbitmap.c | 24 ++++++++++++------------ repair/rt.c | 6 ++---- 4 files changed, 16 insertions(+), 26 deletions(-)
db/check.c +2 -2
@@ -3688,7 +3688,7 @@ process_rtbitmap( }else if (prevbit ==1 ) { len = ((int )bmbno - start_bmbno) * bitsperblock + (bit - start_bit); - log =XFS_RTBLOCKLOG (len); + log =libxfs_highbit64 (len); offs =xfs_rtsumoffs (mp, log, start_bmbno);inc_sumcount (mp, sumcompute, offs); prevbit =0 ;
@@ -3701,7 +3701,7 @@ process_rtbitmap(if (prevbit ==1 ) { len = ((int )bmbno - start_bmbno) * bitsperblock + (bit - start_bit); - log =XFS_RTBLOCKLOG (len); + log =libxfs_highbit64 (len); offs =xfs_rtsumoffs (mp, log, start_bmbno);inc_sumcount (mp, sumcompute, offs); }
libxfs/xfs_format.h +0 -8
@@ -1156,20 +1156,12 @@ static inline bool xfs_dinode_has_large_extent_counts(#define XFS_DFL_RTEXTSIZE (64 * 1024)/* 64kB */ #define XFS_MIN_RTEXTSIZE (4 * 1024)/* 4kB */ -#define XFS_BLOCKSIZE (mp) ((mp)->m_sb.sb_blocksize) -#define XFS_BLOCKMASK (mp) ((mp)->m_blockmask) -/* * RT bit manipulation macros. */ #define XFS_RTMIN (a,b) ((a) < (b) ? (a) : (b))#define XFS_RTMAX (a,b) ((a) > (b) ? (a) : (b)) -#define XFS_RTLOBIT (w) xfs_lowbit32(w) -#define XFS_RTHIBIT (w) xfs_highbit32(w) - -#define XFS_RTBLOCKLOG (b) xfs_highbit64(b) -/* * Dquot and dquot block format definitions */
libxfs/xfs_rtbitmap.c +12 -12
@@ -193,7 +193,7 @@ xfs_rtfind_back(/* * Different. Mark where we are and return. */ - i = bit -XFS_RTHIBIT (wdiff); + i = bit -xfs_highbit32 (wdiff); *rtx = start - i +1 ;return 0 ; }
@@ -231,7 +231,7 @@ xfs_rtfind_back(/* * Different, mark where we are and return. */ - i +=XFS_NBWORD -1 -XFS_RTHIBIT (wdiff); + i +=XFS_NBWORD -1 -xfs_highbit32 (wdiff); *rtx = start - i +1 ;return 0 ; }
@@ -270,7 +270,7 @@ xfs_rtfind_back(/* * Different, mark where we are and return. */ - i +=XFS_NBWORD -1 -XFS_RTHIBIT (wdiff); + i +=XFS_NBWORD -1 -xfs_highbit32 (wdiff); *rtx = start - i +1 ;return 0 ; }else
@@ -346,7 +346,7 @@ xfs_rtfind_forw(/* * Different. Mark where we are and return. */ - i =XFS_RTLOBIT (wdiff) - bit; + i =xfs_lowbit32 (wdiff) - bit; *rtx = start + i -1 ;return 0 ; }
@@ -384,7 +384,7 @@ xfs_rtfind_forw(/* * Different, mark where we are and return. */ - i +=XFS_RTLOBIT (wdiff); + i +=xfs_lowbit32 (wdiff); *rtx = start + i -1 ;return 0 ; }
@@ -421,7 +421,7 @@ xfs_rtfind_forw(/* * Different, mark where we are and return. */ - i +=XFS_RTLOBIT (wdiff); + i +=xfs_lowbit32 (wdiff); *rtx = start + i -1 ;return 0 ; }else
@@ -706,7 +706,7 @@ xfs_rtfree_range(*/ if (preblock < start) { error =xfs_rtmodify_summary (args, -XFS_RTBLOCKLOG (start - preblock), +xfs_highbit64 (start - preblock),xfs_rtx_to_rbmblock (mp, preblock),-1 );if (error) {return error;
@@ -718,7 +718,7 @@ xfs_rtfree_range(*/ if (postblock > end) { error =xfs_rtmodify_summary (args, -XFS_RTBLOCKLOG (postblock - end), +xfs_highbit64 (postblock - end),xfs_rtx_to_rbmblock (mp, end +1 ),-1 );if (error) {return error;
@@ -729,7 +729,7 @@ xfs_rtfree_range(* (new) free extent. */ return xfs_rtmodify_summary (args, -XFS_RTBLOCKLOG (postblock +1 - preblock), +xfs_highbit64 (postblock +1 - preblock),xfs_rtx_to_rbmblock (mp, preblock),1 ); }
@@ -798,7 +798,7 @@ xfs_rtcheck_range(/* * Different, compute first wrong bit and return. */ - i =XFS_RTLOBIT (wdiff) - bit; + i =xfs_lowbit32 (wdiff) - bit; *new = start + i; *stat =0 ;return 0 ;
@@ -837,7 +837,7 @@ xfs_rtcheck_range(/* * Different, compute first wrong bit and return. */ - i +=XFS_RTLOBIT (wdiff); + i +=xfs_lowbit32 (wdiff); *new = start + i; *stat =0 ;return 0 ;
@@ -875,7 +875,7 @@ xfs_rtcheck_range(/* * Different, compute first wrong bit and return. */ - i +=XFS_RTLOBIT (wdiff); + i +=xfs_lowbit32 (wdiff); *new = start + i; *stat =0 ;return 0 ;
repair/rt.c +2 -4
@@ -14,8 +14,6 @@#include "err_protos.h" #include "rt.h" -#define xfs_highbit64 libxfs_highbit64/* for XFS_RTBLOCKLOG macro */ -void rtinit (xfs_mount_t *mp) {
@@ -115,7 +113,7 @@ generate_rtinfo( } }else if (in_extent ==1 ) { len = (int ) (extno - start_ext); - log =XFS_RTBLOCKLOG (len); + log =libxfs_highbit64 (len); offs =xfs_rtsumoffs (mp, log, start_bmbno);inc_sumcount (mp, sumcompute, offs); in_extent =0 ;
@@ -131,7 +129,7 @@ generate_rtinfo( }if (in_extent ==1 ) { len = (int ) (extno - start_ext); - log =XFS_RTBLOCKLOG (len); + log =libxfs_highbit64 (len); offs =xfs_rtsumoffs (mp, log, start_bmbno);inc_sumcount (mp, sumcompute, offs); }
PATCH 53/67 xfs: remove XFS_RTMIN/XFS_RTMAX
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: a39f5ccc30d5a00b7e6d921aa387ad17d1e6d168
Use the kernel min/max helpers instead.
--- libxfs/xfs_format.h | 6 ------ libxfs/xfs_rtbitmap.c | 8 ++++---- mkfs/proto.c | 4 ++-- 3 files changed, 6 insertions(+), 12 deletions(-)
libxfs/xfs_format.h +0 -6
@@ -1156,12 +1156,6 @@ static inline bool xfs_dinode_has_large_extent_counts(#define XFS_DFL_RTEXTSIZE (64 * 1024)/* 64kB */ #define XFS_MIN_RTEXTSIZE (4 * 1024)/* 4kB */ -/* -* RT bit manipulation macros. -*/ -#define XFS_RTMIN (a,b) ((a) < (b) ? (a) : (b)) -#define XFS_RTMAX (a,b) ((a) > (b) ? (a) : (b)) -/* * Dquot and dquot block format definitions */
libxfs/xfs_rtbitmap.c +4 -4
@@ -182,7 +182,7 @@ xfs_rtfind_back(* Calculate first (leftmost) bit number to look at, * and mask for all the relevant bits in this word. */ - firstbit =XFS_RTMAX ((xfs_srtblock_t)(bit - len +1 ),0 ); + firstbit =max_t (xfs_srtblock_t, bit - len +1 ,0 ); mask = (((xfs_rtword_t )1 << (bit - firstbit +1 )) -1 ) << firstbit;/*
@@ -336,7 +336,7 @@ xfs_rtfind_forw(* Calculate last (rightmost) bit number to look at, * and mask for all the relevant bits in this word. */ - lastbit =XFS_RTMIN (bit + len,XFS_NBWORD ); + lastbit =min (bit + len,XFS_NBWORD ); mask = (((xfs_rtword_t )1 << (lastbit - bit)) -1 ) << bit;/* * Calculate the difference between the value there
@@ -571,7 +571,7 @@ xfs_rtmodify_range(/* * Compute first bit not changed and mask of relevant bits. */ - lastbit =XFS_RTMIN (bit + len,XFS_NBWORD ); + lastbit =min (bit + len,XFS_NBWORD ); mask = (((xfs_rtword_t )1 << (lastbit - bit)) -1 ) << bit;/* * Set/clear the active bits.
@@ -785,7 +785,7 @@ xfs_rtcheck_range(/* * Compute first bit not examined. */ - lastbit =XFS_RTMIN (bit + len,XFS_NBWORD ); + lastbit =min (bit + len,XFS_NBWORD );/* * Mask of relevant bits. */
mkfs/proto.c +2 -2
@@ -793,8 +793,8 @@ rtfreesp_init(res_failed (error);libxfs_trans_ijoin (tp, mp->m_rbmip,0 ); - ertx =XFS_RTMIN (mp->m_sb.sb_rextents, - rtx +NBBY * mp->m_sb.sb_blocksize); + ertx =min (mp->m_sb.sb_rextents, + rtx +NBBY * mp->m_sb.sb_blocksize); error = -libxfs_rtfree_extent (tp, rtx, (xfs_rtxlen_t)(ertx - rtx));
PATCH 54/67 xfs: make if_data a void pointer
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 6e145f943bd86be47e54101fa5939f9ed0cb73e5
The xfs_ifork structure currently has a union of the if_root void pointer and the if_data char pointer. In either case it is an opaque pointer that depends on the fork format. Replace the union with a single if_data void pointer as that is what almost all callers want. Only the symlink NULL termination code in xfs_init_local_fork actually needs a new local variable now.
--- db/namei.c | 4 +-- libxfs/util.c | 2 + libxfs/xfs_attr.c | 3 +- libxfs/xfs_attr_leaf.c | 62 +++++++++++++++++-------------------------- libxfs/xfs_bmap.c | 4 +-- libxfs/xfs_dir2.c | 2 + libxfs/xfs_dir2_block.c | 6 +--- libxfs/xfs_dir2_sf.c | 61 +++++++++++++++++------------------------- libxfs/xfs_iext_tree.c | 36 ++++++++++++------------- libxfs/xfs_inode_fork.c | 53 +++++++++++++++++-------------------- libxfs/xfs_inode_fork.h | 8 ++---- libxfs/xfs_symlink_remote.c | 4 +-- repair/phase6.c | 9 +++--- 13 files changed, 107 insertions(+), 147 deletions(-)
db/namei.c +1 -3
@@ -290,14 +290,12 @@ list_sfdir(struct xfs_mount *mp = dp->i_mount;struct xfs_da_geometry *geo = args->geo;struct xfs_dir2_sf_entry *sfep; -struct xfs_dir2_sf_hdr *sfp; +struct xfs_dir2_sf_hdr *sfp = dp->i_df.if_data;xfs_ino_t ino;xfs_dir2_dataptr_t off;unsigned int i;uint8_t filetype; - sfp = (struct xfs_dir2_sf_hdr *)dp->i_df.if_u1.if_data; -/* . and .. entries */ off =xfs_dir2_db_off_to_dataptr (geo, geo->datablk, geo->data_entry_offset);
libxfs/util.c +1 -1
@@ -316,7 +316,7 @@ libxfs_init_new_inode(case S_IFLNK : ip->i_df.if_format =XFS_DINODE_FMT_EXTENTS ; ip->i_df.if_bytes =0 ; - ip->i_df.if_u1.if_root =NULL ; + ip->i_df.if_data =NULL ;break ;default :ASSERT (0 );
libxfs/xfs_attr.c +1 -2
@@ -1047,9 +1047,8 @@ xfs_attr_set(static inline int xfs_attr_sf_totsize (struct xfs_inode *dp) { -struct xfs_attr_shortform *sf; +struct xfs_attr_shortform *sf = dp->i_af.if_data; - sf = (struct xfs_attr_shortform *)dp->i_af.if_u1.if_data;return be16_to_cpu (sf->hdr.totsize); }
libxfs/xfs_attr_leaf.c +24 -38
151 lines not on this page · raw message
libxfs/xfs_bmap.c +2 -2
@@ -741,7 +741,7 @@ xfs_bmap_local_to_extents_empty(ASSERT (ifp->if_nextents ==0 );xfs_bmap_forkoff_reset (ip, whichfork); - ifp->if_u1.if_root =NULL ; + ifp->if_data =NULL ; ifp->if_height =0 ; ifp->if_format =XFS_DINODE_FMT_EXTENTS ;xfs_trans_log_inode (tp, ip,XFS_ILOG_CORE );
@@ -826,7 +826,7 @@ xfs_bmap_local_to_extents(xfs_bmap_local_to_extents_empty (tp, ip, whichfork); flags |=XFS_ILOG_CORE ; - ifp->if_u1.if_root =NULL ; + ifp->if_data =NULL ; ifp->if_height =0 ; rec.br_startoff =0 ;
libxfs/xfs_dir2.c +1 -1
@@ -195,7 +195,7 @@ xfs_dir_isempty(return 1 ;if (dp->i_disk_size >xfs_inode_data_fork_size (dp))return 0 ; - sfp = (xfs_dir2_sf_hdr_t *)dp->i_df.if_u1.if_data; + sfp = dp->i_df.if_data;return !sfp->count; }
libxfs/xfs_dir2_block.c +2 -4
@@ -1086,7 +1086,7 @@ xfs_dir2_sf_to_block(int newoffset;/* offset from current entry */ unsigned int offset = geo->data_entry_offset;xfs_dir2_sf_entry_t *sfep;/* sf entry pointer */ -xfs_dir2_sf_hdr_t *oldsfp;/* old shortform header */ +struct xfs_dir2_sf_hdr *oldsfp = ifp->if_data;xfs_dir2_sf_hdr_t *sfp;/* shortform header */ __be16 *tagp;/* end of data entry */ struct xfs_name name;
@@ -1096,10 +1096,8 @@ xfs_dir2_sf_to_block(ASSERT (ifp->if_format ==XFS_DINODE_FMT_LOCAL );ASSERT (dp->i_disk_size >= offsetof(struct xfs_dir2_sf_hdr , parent)); - oldsfp = (xfs_dir2_sf_hdr_t *)ifp->if_u1.if_data; -ASSERT (ifp->if_bytes == dp->i_disk_size); -ASSERT (ifp->if_u1.if_data !=NULL ); +ASSERT (oldsfp !=NULL );ASSERT (dp->i_disk_size >=xfs_dir2_sf_hdr_size (oldsfp->i8count));ASSERT (dp->i_df.if_nextents ==0 );
libxfs/xfs_dir2_sf.c +24 -37
251 lines not on this page · raw message
libxfs/xfs_iext_tree.c +18 -18
@@ -158,7 +158,7 @@ static void *xfs_iext_find_first_leaf ( struct xfs_ifork *ifp) { -struct xfs_iext_node *node = ifp->if_u1.if_root; +struct xfs_iext_node *node = ifp->if_data;int height;if (!ifp->if_height)
@@ -176,7 +176,7 @@ static void *xfs_iext_find_last_leaf ( struct xfs_ifork *ifp) { -struct xfs_iext_node *node = ifp->if_u1.if_root; +struct xfs_iext_node *node = ifp->if_data;int height, i;if (!ifp->if_height)
@@ -306,7 +306,7 @@ xfs_iext_find_level( xfs_fileoff_t offset, int level) { -struct xfs_iext_node *node = ifp->if_u1.if_root; +struct xfs_iext_node *node = ifp->if_data;int height, i;if (!ifp->if_height)
@@ -402,12 +402,12 @@ xfs_iext_grow(int i;if (ifp->if_height ==1 ) { -struct xfs_iext_leaf *prev = ifp->if_u1.if_root; +struct xfs_iext_leaf *prev = ifp->if_data; node->keys[0 ] =xfs_iext_leaf_key (prev,0 ); node->ptrs[0 ] = prev; }else { -struct xfs_iext_node *prev = ifp->if_u1.if_root; +struct xfs_iext_node *prev = ifp->if_data;ASSERT (ifp->if_height >1 );
@@ -418,7 +418,7 @@ xfs_iext_grow(for (i =1 ; i <KEYS_PER_NODE ; i++) node->keys[i] =XFS_IEXT_KEY_INVALID ; - ifp->if_u1.if_root = node; + ifp->if_data = node; ifp->if_height++; }
@@ -430,7 +430,7 @@ xfs_iext_update_node(int level, void *ptr) { - struct xfs_iext_node *node = ifp->if_u1.if_root; + struct xfs_iext_node *node = ifp->if_data;int height, i;for (height = ifp->if_height; height > level; height--) {
@@ -583,11 +583,11 @@ xfs_iext_alloc_root( {ASSERT (ifp->if_bytes ==0 ); - ifp->if_u1.if_root =kmem_zalloc (sizeof (struct xfs_iext_rec ),KM_NOFS ); + ifp->if_data =kmem_zalloc (sizeof (struct xfs_iext_rec ),KM_NOFS ); ifp->if_height =1 ;/* now that we have a node step into it */ - cur->leaf = ifp->if_u1.if_root; + cur->leaf = ifp->if_data; cur->pos =0 ; }
@@ -603,9 +603,9 @@ xfs_iext_realloc_root(if (new_size /sizeof (struct xfs_iext_rec ) ==RECS_PER_LEAF ) new_size =NODE_SIZE ; - new =krealloc (ifp->if_u1.if_root, new_size,GFP_NOFS | __GFP_NOFAIL); + new =krealloc (ifp->if_data, new_size,GFP_NOFS | __GFP_NOFAIL);memset (new + ifp->if_bytes,0 , new_size - ifp->if_bytes); - ifp->if_u1.if_root = new; + ifp->if_data = new; cur->leaf = new; }
@@ -786,8 +786,8 @@ xfs_iext_remove_node(* If we are at the root and only one entry is left we can just * free this node and update the root pointer. */ -ASSERT (node == ifp->if_u1.if_root); - ifp->if_u1.if_root = node->ptrs[0 ]; +ASSERT (node == ifp->if_data); + ifp->if_data = node->ptrs[0 ]; ifp->if_height--;kmem_free (node); }
@@ -863,8 +863,8 @@ xfs_iext_free_last_leaf( struct xfs_ifork *ifp) { ifp->if_height--; -kmem_free (ifp->if_u1.if_root); - ifp->if_u1.if_root =NULL ; +kmem_free (ifp->if_data); + ifp->if_data =NULL ; }void
@@ -881,7 +881,7 @@ xfs_iext_remove(trace_xfs_iext_remove (ip, cur, state, _RET_IP_);ASSERT (ifp->if_height >0 ); -ASSERT (ifp->if_u1.if_root !=NULL ); +ASSERT (ifp->if_data !=NULL );ASSERT (xfs_iext_valid (ifp, cur));xfs_iext_inc_seq (ifp);
@@ -1051,9 +1051,9 @@ voidxfs_iext_destroy ( struct xfs_ifork *ifp) { -xfs_iext_destroy_node (ifp->if_u1.if_root, ifp->if_height); +xfs_iext_destroy_node (ifp->if_data, ifp->if_height); ifp->if_bytes =0 ; ifp->if_height =0 ; - ifp->if_u1.if_root =NULL ; + ifp->if_data =NULL ; }
libxfs/xfs_inode_fork.c +24 -29
125 lines not on this page · raw message
libxfs/xfs_inode_fork.h +3 -5
@@ -13,14 +13,12 @@ struct xfs_dinode;* File incore extent information, present for each of data & attr forks. */ struct xfs_ifork { -int64_t if_bytes;/* bytes in if_u1 */ +int64_t if_bytes;/* bytes in if_data */ struct xfs_btree_block *if_broot;/* file's incore btree root */ unsigned int if_seq;/* fork mod counter */ int if_height;/* height of the extent tree */ -union { -void *if_root;/* extent tree root */ -char *if_data;/* inline file data */ - } if_u1; +void *if_data;/* extent tree root or +inline data */ xfs_extnum_t if_nextents;/* # of extents in this fork */ short if_broot_bytes;/* bytes allocated for root */ int8_t if_format;/* format of this fork */
libxfs/xfs_symlink_remote.c +2 -2
@@ -172,7 +172,7 @@ xfs_symlink_local_to_remote(if (!xfs_has_crc (mp)) { bp->b_ops =NULL ; -memcpy (bp->b_addr, ifp->if_u1.if_data, ifp->if_bytes); +memcpy (bp->b_addr, ifp->if_data, ifp->if_bytes);xfs_trans_log_buf (tp, bp,0 , ifp->if_bytes -1 );return ; }
@@ -188,7 +188,7 @@ xfs_symlink_local_to_remote( buf = bp->b_addr; buf +=xfs_symlink_hdr_set (mp, ip->i_ino,0 , ifp->if_bytes, bp); -memcpy (buf, ifp->if_u1.if_data, ifp->if_bytes); +memcpy (buf, ifp->if_data, ifp->if_bytes);xfs_trans_log_buf (tp, bp,0 ,sizeof (struct xfs_dsymlink_hdr ) + ifp->if_bytes -1 ); }
repair/phase6.c +4 -5
@@ -515,7 +515,7 @@ mk_rbmino(xfs_mount_t *mp)* now the ifork */ ip->i_df.if_bytes =0 ; - ip->i_df.if_u1.if_root =NULL ; + ip->i_df.if_data =NULL ; ip->i_disk_size = mp->m_sb.sb_rbmblocks * mp->m_sb.sb_blocksize;
@@ -770,7 +770,7 @@ mk_rsumino(xfs_mount_t *mp)* now the ifork */ ip->i_df.if_bytes =0 ; - ip->i_df.if_u1.if_root =NULL ; + ip->i_df.if_data =NULL ; ip->i_disk_size = mp->m_rsumsize;
@@ -870,7 +870,7 @@ mk_root_dir(xfs_mount_t *mp)* now the ifork */ ip->i_df.if_bytes =0 ; - ip->i_df.if_u1.if_root =NULL ; + ip->i_df.if_data =NULL ;/* * initialize the directory
@@ -2472,7 +2472,7 @@ shortform_dir2_entry_check( {xfs_ino_t lino;xfs_ino_t parent; -struct xfs_dir2_sf_hdr *sfp; +struct xfs_dir2_sf_hdr *sfp = ip->i_df.if_data;struct xfs_dir2_sf_entry *sfep;struct xfs_dir2_sf_entry *next_sfep;struct xfs_ifork *ifp;
@@ -2487,7 +2487,6 @@ shortform_dir2_entry_check(int i8; ifp = &ip->i_df; - sfp = (struct xfs_dir2_sf_hdr *) ifp->if_u1.if_data; *ino_dirty =0 ; bytes_deleted =0 ;
PATCH 55/67 xfs: return if_data from xfs_idata_realloc
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 45c76a2add55b332d965c901e14004ae0134a67e
Many of the xfs_idata_realloc callers need to set a local pointer to the just reallocated if_data memory. Return the pointer to simplify them a bit and use the opportunity to re-use krealloc for freeing if_data if the size hits 0.
--- libxfs/xfs_attr_leaf.c | 7 +++---- libxfs/xfs_dir2_sf.c | 25 ++++++++++--------------- libxfs/xfs_inode_fork.c | 20 ++++++++------------ libxfs/xfs_inode_fork.h | 2 +- 4 files changed, 22 insertions(+), 32 deletions(-)
libxfs/xfs_attr_leaf.c +3 -4
@@ -687,8 +687,8 @@ xfs_attr_shortform_create(ASSERT (ifp->if_bytes ==0 );if (ifp->if_format ==XFS_DINODE_FMT_EXTENTS ) ifp->if_format =XFS_DINODE_FMT_LOCAL ; -xfs_idata_realloc (dp,sizeof (*hdr),XFS_ATTR_FORK ); - hdr = ifp->if_data; + + hdr =xfs_idata_realloc (dp,sizeof (*hdr),XFS_ATTR_FORK );memset (hdr,0 ,sizeof (*hdr)); hdr->totsize =cpu_to_be16 (sizeof (*hdr));xfs_trans_log_inode (args->trans, dp,XFS_ILOG_CORE |XFS_ILOG_ADATA );
@@ -764,8 +764,7 @@ xfs_attr_shortform_add( offset = (char *)sfe - (char *)sf; size =xfs_attr_sf_entsize_byname (args->namelen, args->valuelen); -xfs_idata_realloc (dp, size,XFS_ATTR_FORK ); - sf = ifp->if_data; + sf =xfs_idata_realloc (dp, size,XFS_ATTR_FORK ); sfe = (struct xfs_attr_sf_entry *)((char *)sf + offset); sfe->namelen = args->namelen;
libxfs/xfs_dir2_sf.c +10 -15
@@ -466,12 +466,11 @@ xfs_dir2_sf_addname_easy(/* * Grow the in-inode space. */ -xfs_idata_realloc (dp,xfs_dir2_sf_entsize (mp, sfp, args->namelen), + sfp =xfs_idata_realloc (dp,xfs_dir2_sf_entsize (mp, sfp, args->namelen),XFS_DATA_FORK );/* * Need to set up again due to realloc of the inode data. */ - sfp = dp->i_df.if_data; sfep = (xfs_dir2_sf_entry_t *)((char *)sfp + byteoff);/* * Fill in the new entry.
@@ -551,11 +550,8 @@ xfs_dir2_sf_addname_hard(* the data. */ xfs_idata_realloc (dp, -old_isize,XFS_DATA_FORK ); -xfs_idata_realloc (dp, new_isize,XFS_DATA_FORK ); -/* -* Reset the pointer since the buffer was reallocated. -*/ - sfp = dp->i_df.if_data; + sfp =xfs_idata_realloc (dp, new_isize,XFS_DATA_FORK ); +/* * Copy the first part of the directory, including the header. */
@@ -820,15 +816,13 @@ xfs_dir2_sf_create(ASSERT (dp->i_df.if_bytes ==0 ); i8count = pino >XFS_DIR2_MAX_SHORT_INUM ; size =xfs_dir2_sf_hdr_size (i8count); +/* -* Make a buffer for the data. +* Make a buffer for the data and fill in the header. */ -xfs_idata_realloc (dp, size,XFS_DATA_FORK ); -/* -* Fill in the header, -*/ - sfp = dp->i_df.if_data; + sfp =xfs_idata_realloc (dp, size,XFS_DATA_FORK ); sfp->i8count = i8count; +/* * Now can put in the inode number, since i8count is set. */
@@ -976,11 +970,12 @@ xfs_dir2_sf_removename(*/ sfp->count--; dp->i_disk_size = newsize; +/* * Reallocate, making it smaller. */ -xfs_idata_realloc (dp, newsize - oldsize,XFS_DATA_FORK ); - sfp = dp->i_df.if_data; + sfp =xfs_idata_realloc (dp, newsize - oldsize,XFS_DATA_FORK ); +/* * Are we changing inode number size? */
libxfs/xfs_inode_fork.c +8 -12
@@ -494,7 +494,7 @@ xfs_iroot_realloc(* byte_diff -- the change in the number of bytes, positive or negative, * requested for the if_data array. */ -void +void * xfs_idata_realloc( struct xfs_inode *ip, int64_t byte_diff,
@@ -506,19 +506,15 @@ xfs_idata_realloc(ASSERT (new_size >=0 );ASSERT (new_size <=xfs_inode_fork_size (ip, whichfork)); -if (byte_diff ==0 ) -return ; - -if (new_size ==0 ) { -kmem_free (ifp->if_data); - ifp->if_data =NULL ; - ifp->if_bytes =0 ; -return ; +if (byte_diff) { + ifp->if_data =krealloc (ifp->if_data, new_size, +GFP_NOFS | __GFP_NOFAIL); +if (new_size ==0 ) + ifp->if_data =NULL ; + ifp->if_bytes = new_size; } - ifp->if_data =krealloc (ifp->if_data, new_size, -GFP_NOFS | __GFP_NOFAIL); - ifp->if_bytes = new_size; +return ifp->if_data; }/* Free all memory and reset a fork back to its initial state. */
libxfs/xfs_inode_fork.h +1 -1
@@ -168,7 +168,7 @@ int xfs_iformat_attr_fork(struct xfs_inode *, struct xfs_dinode *);void xfs_iflush_fork (struct xfs_inode *,struct xfs_dinode *,struct xfs_inode_log_item *,int );void xfs_idestroy_fork (struct xfs_ifork *ifp); -void xfs_idata_realloc (struct xfs_inode *ip,int64_t byte_diff, +void *xfs_idata_realloc (struct xfs_inode *ip,int64_t byte_diff,int whichfork);void xfs_iroot_realloc (struct xfs_inode *,int ,int );int xfs_iread_extents (struct xfs_trans *,struct xfs_inode *,int );
PATCH 56/67 xfs: move the xfs_attr_sf_lookup tracepoint
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 14f2e4ab5d0310c2bb231941d9884fa5bae47fab
trace_xfs_attr_sf_lookup is currently only called by xfs_attr_shortform_lookup, which despit it's name is a simple helper for xfs_attr_shortform_addname, which has it's own tracing. Move the callsite to xfs_attr_shortform_getvalue, which is the closest thing to a high level lookup we have for the Linux xattr API.
--- libxfs/xfs_attr_leaf.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
libxfs/xfs_attr_leaf.c +3 -2
@@ -873,8 +873,6 @@ xfs_attr_shortform_lookup(struct xfs_attr_sf_entry *sfe;int i; -trace_xfs_attr_sf_lookup (args); -ASSERT (ifp->if_format ==XFS_DINODE_FMT_LOCAL ); sfe = &sf->list[0 ]; for (i = 0; i < sf->hdr.count;
@@ -902,6 +900,9 @@ xfs_attr_shortform_getvalue(int i;ASSERT (args->dp->i_af.if_format ==XFS_DINODE_FMT_LOCAL ); + +trace_xfs_attr_sf_lookup (args); + sfe = &sf->list[0 ];for (i =0 ; i < sf->hdr.count; sfe =xfs_attr_sf_nextentry (sfe), i++) {
PATCH 57/67 xfs: simplify xfs_attr_sf_findname
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 6c8d169bbd51fc10d1d0029d495962881315b4c2
xfs_attr_sf_findname has the simple job of finding a xfs_attr_sf_entry in the attr fork, but the convoluted calling convention obfuscates that.
Return the found entry as the return value instead of an pointer argument, as the -ENOATTR/-EEXIST can be trivally derived from that, and remove the basep argument, as it is equivalent of the offset of sfe in the data for if an sfe was found, or an offset of totsize if not was found. To simplify the totsize computation add a xfs_attr_sf_endptr helper that returns the imaginative xfs_attr_sf_entry at the end of the current attrs.
--- libxfs/xfs_attr.c | 7 ++- libxfs/xfs_attr_leaf.c | 102 ++++++++++++++++++------------------------------ libxfs/xfs_attr_leaf.h | 4 -- libxfs/xfs_attr_sf.h | 7 +++ 4 files changed, 51 insertions(+), 69 deletions(-)
libxfs/xfs_attr.c +5 -2
@@ -860,8 +860,11 @@ xfs_attr_lookup(if (!xfs_inode_hasattr (dp))return -ENOATTR ; -if (dp->i_af.if_format ==XFS_DINODE_FMT_LOCAL ) -return xfs_attr_sf_findname (args,NULL ,NULL ); +if (dp->i_af.if_format ==XFS_DINODE_FMT_LOCAL ) { +if (xfs_attr_sf_findname (args)) +return -EEXIST ; +return -ENOATTR ; + }if (xfs_attr_is_leaf (dp)) { error =xfs_attr_leaf_hasname (args, &bp);
libxfs/xfs_attr_leaf.c +38 -64
145 lines not on this page · raw message
libxfs/xfs_attr_leaf.h +1 -3
@@ -51,9 +51,7 @@ int xfs_attr_shortform_lookup(struct xfs_da_args *args);int xfs_attr_shortform_getvalue (struct xfs_da_args *args);int xfs_attr_shortform_to_leaf (struct xfs_da_args *args);int xfs_attr_sf_removename (struct xfs_da_args *args); -int xfs_attr_sf_findname (struct xfs_da_args *args, -struct xfs_attr_sf_entry **sfep, -unsigned int *basep); +struct xfs_attr_sf_entry *xfs_attr_sf_findname (struct xfs_da_args *args);int xfs_attr_shortform_allfit (struct xfs_buf *bp,struct xfs_inode *dp);int xfs_attr_shortform_bytesfit (struct xfs_inode *dp,int bytes); xfs_failaddr_t xfs_attr_shortform_verify(struct xfs_attr_shortform *sfp,
libxfs/xfs_attr_sf.h +7 -0
@@ -48,4 +48,11 @@ xfs_attr_sf_nextentry(struct xfs_attr_sf_entry *sfep) return (void *)sfep + xfs_attr_sf_entsize(sfep); } +/* pointer to the space after the last entry, e.g. for adding a new one */ +static inline struct xfs_attr_sf_entry * +xfs_attr_sf_endptr (struct xfs_attr_shortform *sf) +{ +return (void *)sf +be16_to_cpu (sf->hdr.totsize); +} +#endif /* __XFS_ATTR_SF_H__ */
PATCH 58/67 xfs: remove xfs_attr_shortform_lookup
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 22b7b1f597a6a21fb7b3791a55f3a7ae54d2dfe4
xfs_attr_shortform_lookup is only used by xfs_attr_shortform_addname, which is much better served by calling xfs_attr_sf_findname. Switch it over and remove xfs_attr_shortform_lookup.
--- libxfs/xfs_attr.c | 21 +++++++-------------- libxfs/xfs_attr_leaf.c | 24 ------------------------ libxfs/xfs_attr_leaf.h | 1 - 3 files changed, 7 insertions(+), 39 deletions(-)
libxfs/xfs_attr.c +7 -14
@@ -1064,19 +1064,14 @@ xfs_attr_shortform_addname( struct xfs_da_args *args) {int newsize, forkoff; -int error;trace_xfs_attr_sf_addname (args); - error =xfs_attr_shortform_lookup (args); -switch (error) { -case -ENOATTR : -if (args->op_flags &XFS_DA_OP_REPLACE ) -return error; -break ; -case -EEXIST : +if (xfs_attr_sf_findname (args)) { +int error; +if (!(args->op_flags &XFS_DA_OP_REPLACE )) -return error; +return -EEXIST ; error =xfs_attr_sf_removename (args);if (error)
@@ -1089,11 +1084,9 @@ xfs_attr_shortform_addname(* around. */ args->op_flags &= ~XFS_DA_OP_REPLACE ; -break ; -case 0 : -break ; -default : -return error; + }else { +if (args->op_flags &XFS_DA_OP_REPLACE ) +return -ENOATTR ; } if (args->namelen >= XFS_ATTR_SF_ENTSIZE_MAX ||
libxfs/xfs_attr_leaf.c +0 -24
@@ -834,30 +834,6 @@ xfs_attr_sf_removename(return 0 ; } -/* -* Look up a name in a shortform attribute list structure. -*/ -/*ARGSUSED*/ -int -xfs_attr_shortform_lookup ( -struct xfs_da_args *args) -{ -struct xfs_ifork *ifp = &args->dp->i_af; -struct xfs_attr_shortform *sf = ifp->if_data; -struct xfs_attr_sf_entry *sfe; -int i; - -ASSERT (ifp->if_format ==XFS_DINODE_FMT_LOCAL ); - sfe = &sf->list[0 ]; -for (i =0 ; i < sf->hdr.count; - sfe =xfs_attr_sf_nextentry (sfe), i++) { -if (xfs_attr_match (args, sfe->namelen, sfe->nameval, - sfe->flags)) -return -EEXIST ; - } -return -ENOATTR ; -} -/* * Retrieve the attribute value and length. *
libxfs/xfs_attr_leaf.h +0 -1
@@ -47,7 +47,6 @@ struct xfs_attr3_icleaf_hdr {*/ void xfs_attr_shortform_create (struct xfs_da_args *args);void xfs_attr_shortform_add (struct xfs_da_args *args,int forkoff); -int xfs_attr_shortform_lookup (struct xfs_da_args *args);int xfs_attr_shortform_getvalue (struct xfs_da_args *args);int xfs_attr_shortform_to_leaf (struct xfs_da_args *args);int xfs_attr_sf_removename (struct xfs_da_args *args);
PATCH 59/67 xfs: use xfs_attr_sf_findname in xfs_attr_shortform_getvalue
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 1fb4b0def7b5a5bf91ad62a112d8d3f6dc76585f
xfs_attr_shortform_getvalue duplicates the logic in xfs_attr_sf_findname. Use the helper instead.
--- libxfs/xfs_attr_leaf.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
libxfs/xfs_attr_leaf.c +5 -11
@@ -845,23 +845,17 @@ intxfs_attr_shortform_getvalue ( struct xfs_da_args *args) { -struct xfs_attr_shortform *sf = args->dp->i_af.if_data;struct xfs_attr_sf_entry *sfe; -int i;ASSERT (args->dp->i_af.if_format ==XFS_DINODE_FMT_LOCAL );trace_xfs_attr_sf_lookup (args); - sfe = &sf->list[0 ]; -for (i =0 ; i < sf->hdr.count; - sfe =xfs_attr_sf_nextentry (sfe), i++) { -if (xfs_attr_match (args, sfe->namelen, sfe->nameval, - sfe->flags)) -return xfs_attr_copy_value (args, - &sfe->nameval[args->namelen], sfe->valuelen); - } -return -ENOATTR ; + sfe =xfs_attr_sf_findname (args); +if (!sfe) +return -ENOATTR ; +return xfs_attr_copy_value (args, &sfe->nameval[args->namelen], + sfe->valuelen); }/* Convert from using the shortform to the leaf format. */
PATCH 60/67 xfs: remove struct xfs_attr_shortform
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 414147225400a0c4562ebfb0fdd40f065099ede4
sparse complains about struct xfs_attr_shortform because it embeds a structure with a variable sized array in a variable sized array.
Given that xfs_attr_shortform is not a very useful structure, and the dir2 equivalent has been removed a long time ago, remove it as well.
Provide a xfs_attr_sf_firstentry helper that returns the first xfs_attr_sf_entry behind a xfs_attr_sf_hdr to replace the structure dereference.
--- db/attrshort.c | 33 ++++++++++++++++----------------- db/check.c | 8 ++++---- db/inode.c | 6 +++--- db/metadump.c | 16 +++++++--------- libxfs/libxfs_api_defs.h | 2 ++ libxfs/xfs_attr.c | 4 ++-- libxfs/xfs_attr_leaf.c | 37 ++++++++++++++++++------------------- libxfs/xfs_attr_leaf.h | 2 +- libxfs/xfs_attr_sf.h | 13 ++++++++++--- libxfs/xfs_da_format.h | 31 ++++++++++++++++++------------- libxfs/xfs_inode_fork.c | 5 ++--- libxfs/xfs_ondisk.h | 14 +++++++------- repair/attr_repair.c | 44 ++++++++++++++++++++------------------------ repair/dinode.c | 21 +++++++++++---------- 14 files changed, 121 insertions(+), 115 deletions(-)
db/attrshort.c +16 -17
@@ -18,9 +18,8 @@ static int attr_sf_entry_value_offset(void *obj, int startoff, int idx);static int attr_shortform_list_count (void *obj,int startoff);static int attr_shortform_list_offset (void *obj,int startoff,int idx); -#define OFF (f) bitize(offsetof(struct xfs_attr_shortform, f))const field_t attr_shortform_flds[] = { - {"hdr" ,FLDT_ATTR_SF_HDR ,OI (OFF (hdr)),C1 ,0 ,TYP_NONE }, + {"hdr" ,FLDT_ATTR_SF_HDR ,OI (0 ),C1 ,0 ,TYP_NONE }, {"list" ,FLDT_ATTR_SF_ENTRY , attr_shortform_list_offset, attr_shortform_list_count,FLD_ARRAY |FLD_COUNT |FLD_OFFSET ,TYP_NONE }, {NULL }
@@ -71,11 +70,11 @@ attr_sf_entry_size( {struct xfs_attr_sf_entry *e;int i; -struct xfs_attr_shortform *sf; +struct xfs_attr_sf_hdr *hdr;ASSERT (bitoffs (startoff) ==0 ); - sf = (struct xfs_attr_shortform *)((char *)obj +byteize (startoff)); - e = &sf->list[0 ]; + hdr = (struct xfs_attr_sf_hdr *)((char *)obj +byteize (startoff)); + e =libxfs_attr_sf_firstentry (hdr);for (i =0 ; i < idx; i++) e =xfs_attr_sf_nextentry (e);return bitize ((int )xfs_attr_sf_entsize (e));
@@ -113,11 +112,11 @@ attr_shortform_list_count( void *obj, int startoff) { -struct xfs_attr_shortform *sf; +struct xfs_attr_sf_hdr *hdr;ASSERT (bitoffs (startoff) ==0 ); - sf = (struct xfs_attr_shortform *)((char *)obj +byteize (startoff)); -return sf->hdr.count; + hdr = (struct xfs_attr_sf_hdr *)((char *)obj +byteize (startoff)); +return hdr->count; } static int
@@ -128,14 +127,14 @@ attr_shortform_list_offset( {struct xfs_attr_sf_entry *e;int i; -struct xfs_attr_shortform *sf; +struct xfs_attr_sf_hdr *hdr;ASSERT (bitoffs (startoff) ==0 ); - sf = (struct xfs_attr_shortform *)((char *)obj +byteize (startoff)); - e = &sf->list[0 ]; + hdr = (struct xfs_attr_sf_hdr *)((char *)obj +byteize (startoff)); + e =libxfs_attr_sf_firstentry (hdr);for (i =0 ; i < idx; i++) e =xfs_attr_sf_nextentry (e); -return bitize ((int )((char *)e - (char *)sf)); +return bitize ((int )((char *)e - (char *)hdr)); }/*ARGSUSED*/
@@ -147,13 +146,13 @@ attrshort_size( {struct xfs_attr_sf_entry *e;int i; -struct xfs_attr_shortform *sf; +struct xfs_attr_sf_hdr *hdr;ASSERT (bitoffs (startoff) ==0 );ASSERT (idx ==0 ); - sf = (struct xfs_attr_shortform *)((char *)obj +byteize (startoff)); - e = &sf->list[0 ]; -for (i =0 ; i < sf->hdr.count; i++) + hdr = (struct xfs_attr_sf_hdr *)((char *)obj +byteize (startoff)); + e =libxfs_attr_sf_firstentry (hdr); +for (i =0 ; i < hdr->count; i++) e =xfs_attr_sf_nextentry (e); -return bitize ((int )((char *)e - (char *)sf)); +return bitize ((int )((char *)e - (char *)hdr)); }
db/check.c +4 -4
@@ -3069,7 +3069,7 @@ process_lclinode( blkmap_t **blkmapp, int whichfork) { -struct xfs_attr_shortform *asf; +struct xfs_attr_sf_hdr *hdr;xfs_fsblock_t bno; bno =XFS_INO_TO_FSB (mp, id->ino);
@@ -3082,12 +3082,12 @@ process_lclinode( error++; }else if (whichfork ==XFS_ATTR_FORK ) { - asf = (struct xfs_attr_shortform *)XFS_DFORK_APTR (dip); -if (be16_to_cpu (asf->hdr.totsize) >XFS_DFORK_ASIZE (dip, mp)) { + hdr =XFS_DFORK_APTR (dip); +if (be16_to_cpu (hdr->totsize) >XFS_DFORK_ASIZE (dip, mp)) {if (!sflag || id->ilist ||CHECK_BLIST (bno))dbprintf (_ ("local inode %lld attr is too large " "(size %d)\n" ), - id->ino,be16_to_cpu (asf->hdr.totsize)); + id->ino,be16_to_cpu (hdr->totsize)); error++; } }
db/inode.c +3 -3
@@ -356,7 +356,7 @@ inode_a_size( int startoff, int idx) { -struct xfs_attr_shortform *asf; +struct xfs_attr_sf_hdr *hdr;struct xfs_dinode *dip;ASSERT (startoff ==0 );
@@ -364,8 +364,8 @@ inode_a_size( dip = obj;switch (dip->di_aformat) {case XFS_DINODE_FMT_LOCAL : - asf = (struct xfs_attr_shortform *)XFS_DFORK_APTR (dip); -return bitize (be16_to_cpu (asf->hdr.totsize)); + hdr =XFS_DFORK_APTR (dip); +return bitize (be16_to_cpu (hdr->totsize));case XFS_DINODE_FMT_EXTENTS :return (int )xfs_dfork_attr_extents (dip) *bitsz (xfs_bmbt_rec_t);case XFS_DINODE_FMT_BTREE :
db/metadump.c +7 -9
@@ -1035,16 +1035,15 @@ process_sf_attr(* values with 'v' (to see a valid string length, as opposed to NULLs) */ -struct xfs_attr_shortform *asfp; -struct xfs_attr_sf_entry *asfep; +struct xfs_attr_sf_hdr *hdr =XFS_DFORK_APTR (dip); +struct xfs_attr_sf_entry *asfep =libxfs_attr_sf_firstentry (hdr);int ino_attr_size;int i; - asfp = (struct xfs_attr_shortform *)XFS_DFORK_APTR (dip); -if (asfp->hdr.count ==0 ) +if (hdr->count ==0 )return ; - ino_attr_size =be16_to_cpu (asfp->hdr.totsize); + ino_attr_size =be16_to_cpu (hdr->totsize);if (ino_attr_size >XFS_DFORK_ASIZE (dip, mp)) { ino_attr_size =XFS_DFORK_ASIZE (dip, mp);if (metadump.show_warnings)
@@ -1052,9 +1051,8 @@ process_sf_attr( (long long)metadump.cur_ino); } - asfep = &asfp->list[0 ]; -for (i =0 ; (i < asfp->hdr.count) && - ((char *)asfep - (char *)asfp < ino_attr_size); i++) { +for (i =0 ; (i < hdr->count) && + ((char *)asfep - (char *)hdr < ino_attr_size); i++) {int namelen = asfep->namelen;
@@ -1063,7 +1061,7 @@ process_sf_attr(print_warning ("zero length attr entry in inode " "%llu" , (long long )metadump.cur_ino);break ; - }else if ((char *)asfep - (char *)asfp + + }else if ((char *)asfep - (char *)hdr +xfs_attr_sf_entsize (asfep) > ino_attr_size) {if (metadump.show_warnings) print_warning("attr entry length in inode %llu "
libxfs/libxfs_api_defs.h +2 -0
@@ -40,6 +40,8 @@#define xfs_attr_leaf_newentsize libxfs_attr_leaf_newentsize#define xfs_attr_namecheck libxfs_attr_namecheck#define xfs_attr_set libxfs_attr_set +#define xfs_attr_sf_firstentry libxfs_attr_sf_firstentry +#define xfs_attr_shortform_verify libxfs_attr_shortform_verify#define __xfs_bmap_add_free __libxfs_bmap_add_free#define xfs_bmapi_read libxfs_bmapi_read
libxfs/xfs_attr.c +2 -2
@@ -1050,9 +1050,9 @@ xfs_attr_set(static inline int xfs_attr_sf_totsize (struct xfs_inode *dp) { -struct xfs_attr_shortform *sf = dp->i_af.if_data; +struct xfs_attr_sf_hdr *sf = dp->i_af.if_data; -return be16_to_cpu (sf->hdr.totsize); +return be16_to_cpu (sf->totsize); }/*
libxfs/xfs_attr_leaf.c +18 -19
@@ -701,10 +701,10 @@ struct xfs_attr_sf_entry *xfs_attr_sf_findname ( struct xfs_da_args *args) { -struct xfs_attr_shortform *sf = args->dp->i_af.if_data; +struct xfs_attr_sf_hdr *sf = args->dp->i_af.if_data;struct xfs_attr_sf_entry *sfe; -for (sfe = &sf->list[0 ]; +for (sfe =xfs_attr_sf_firstentry (sf); sfe <xfs_attr_sf_endptr (sf); sfe =xfs_attr_sf_nextentry (sfe)) { if (xfs_attr_match(args, sfe->namelen, sfe->nameval,
@@ -727,7 +727,7 @@ xfs_attr_shortform_add(struct xfs_inode *dp = args->dp;struct xfs_mount *mp = dp->i_mount;struct xfs_ifork *ifp = &dp->i_af; -struct xfs_attr_shortform *sf = ifp->if_data; +struct xfs_attr_sf_hdr *sf = ifp->if_data;struct xfs_attr_sf_entry *sfe;int size;
@@ -747,8 +747,8 @@ xfs_attr_shortform_add( sfe->flags = args->attr_filter;memcpy (sfe->nameval, args->name, args->namelen);memcpy (&sfe->nameval[args->namelen], args->value, args->valuelen); - sf->hdr.count++; -be16_add_cpu (&sf->hdr.totsize, size); + sf->count++; +be16_add_cpu (&sf->totsize, size);xfs_trans_log_inode (args->trans, dp,XFS_ILOG_CORE |XFS_ILOG_ADATA );xfs_sbversion_add_attr2 (mp, args->trans);
@@ -779,9 +779,9 @@ xfs_attr_sf_removename( {struct xfs_inode *dp = args->dp;struct xfs_mount *mp = dp->i_mount; -struct xfs_attr_shortform *sf = dp->i_af.if_data; +struct xfs_attr_sf_hdr *sf = dp->i_af.if_data;struct xfs_attr_sf_entry *sfe; -uint16_t totsize =be16_to_cpu (sf->hdr.totsize); +uint16_t totsize =be16_to_cpu (sf->totsize);void *next, *end;int size =0 ;
@@ -806,9 +806,9 @@ xfs_attr_sf_removename( end =xfs_attr_sf_endptr (sf);if (next < end)memmove (sfe, next, end - next); - sf->hdr.count--; + sf->count--; totsize -= size; - sf->hdr.totsize =cpu_to_be16 (totsize); + sf->totsize =cpu_to_be16 (totsize);/* * Fix up the start offset of the attribute fork
@@ -865,21 +865,21 @@ xfs_attr_shortform_to_leaf( {struct xfs_inode *dp = args->dp;struct xfs_ifork *ifp = &dp->i_af; -struct xfs_attr_shortform *sf = ifp->if_data; +struct xfs_attr_sf_hdr *sf = ifp->if_data;struct xfs_attr_sf_entry *sfe; +int size =be16_to_cpu (sf->totsize);struct xfs_da_args nargs;char *tmpbuffer; -int error, i, size; +int error, i;xfs_dablk_t blkno;struct xfs_buf *bp;trace_xfs_attr_sf_to_leaf (args); - size =be16_to_cpu (sf->hdr.totsize); tmpbuffer =kmem_alloc (size,0 );ASSERT (tmpbuffer !=NULL );memcpy (tmpbuffer, ifp->if_data, size); - sf = (struct xfs_attr_shortform *)tmpbuffer; + sf = (struct xfs_attr_sf_hdr *)tmpbuffer;xfs_idata_realloc (dp, -size,XFS_ATTR_FORK );xfs_bmap_local_to_extents_empty (args->trans, dp,XFS_ATTR_FORK );
@@ -902,8 +902,8 @@ xfs_attr_shortform_to_leaf( nargs.trans = args->trans; nargs.op_flags =XFS_DA_OP_OKNOENT ; - sfe = &sf->list[0 ]; -for (i =0 ; i < sf->hdr.count; i++) { + sfe =xfs_attr_sf_firstentry (sf); +for (i =0 ; i < sf->count; i++) { nargs.name = sfe->nameval; nargs.namelen = sfe->namelen; nargs.value = &sfe->nameval[nargs.namelen];
@@ -970,10 +970,10 @@ xfs_attr_shortform_allfit(/* Verify the consistency of a raw inline attribute fork. */ xfs_failaddr_t xfs_attr_shortform_verify ( -struct xfs_attr_shortform *sfp, +struct xfs_attr_sf_hdr *sfp,size_t size) { -struct xfs_attr_sf_entry *sfep; +struct xfs_attr_sf_entry *sfep =xfs_attr_sf_firstentry (sfp);struct xfs_attr_sf_entry *next_sfep;char *endp;int i;
@@ -987,8 +987,7 @@ xfs_attr_shortform_verify( endp = (char *)sfp + size;/* Check all reported entries */ - sfep = &sfp->list[0 ]; -for (i =0 ; i < sfp->hdr.count; i++) { +for (i =0 ; i < sfp->count; i++) {/* * struct xfs_attr_sf_entry has a variable length. * Check the fixed-offset parts of the structure are
libxfs/xfs_attr_leaf.h +1 -1
@@ -53,7 +53,7 @@ int xfs_attr_sf_removename(struct xfs_da_args *args);struct xfs_attr_sf_entry *xfs_attr_sf_findname (struct xfs_da_args *args);int xfs_attr_shortform_allfit (struct xfs_buf *bp,struct xfs_inode *dp);int xfs_attr_shortform_bytesfit (struct xfs_inode *dp,int bytes); -xfs_failaddr_t xfs_attr_shortform_verify (struct xfs_attr_shortform *sfp, +xfs_failaddr_t xfs_attr_shortform_verify (struct xfs_attr_sf_hdr *sfp,size_t size);void xfs_attr_fork_remove (struct xfs_inode *ip,struct xfs_trans *tp);
libxfs/xfs_attr_sf.h +10 -3
@@ -41,7 +41,14 @@ static inline int xfs_attr_sf_entsize(struct xfs_attr_sf_entry *sfep)return struct_size (sfep, nameval, sfep->namelen + sfep->valuelen); } -/* next entry in struct */ +/* first entry in the SF attr fork */ +static inline struct xfs_attr_sf_entry * +xfs_attr_sf_firstentry (struct xfs_attr_sf_hdr *hdr) +{ +return (struct xfs_attr_sf_entry *)(hdr +1 ); +} + +/* next entry after sfep */ static inline struct xfs_attr_sf_entry *xfs_attr_sf_nextentry (struct xfs_attr_sf_entry *sfep) {
@@ -50,9 +57,9 @@ xfs_attr_sf_nextentry(struct xfs_attr_sf_entry *sfep)/* pointer to the space after the last entry, e.g. for adding a new one */ static inline struct xfs_attr_sf_entry * -xfs_attr_sf_endptr (struct xfs_attr_shortform *sf) +xfs_attr_sf_endptr (struct xfs_attr_sf_hdr *sf) { -return (void *)sf +be16_to_cpu (sf->hdr.totsize); +return (void *)sf +be16_to_cpu (sf->totsize); }#endif /* __XFS_ATTR_SF_H__ */
libxfs/xfs_da_format.h +18 -13
@@ -578,20 +578,25 @@ xfs_dir2_block_leaf_p(struct xfs_dir2_block_tail *btp)#define XFS_ATTR_LEAF_MAPSIZE 3/* how many freespace slots */ /* -* Entries are packed toward the top as tight as possible. +* Attribute storage when stored inside the inode. +* +* Small attribute lists are packed as tightly as possible so as to fit into the +* literal area of the inode. +* +* These "shortform" attribute forks consist of a single xfs_attr_sf_hdr header +* followed by zero or more xfs_attr_sf_entry structures. */ -struct xfs_attr_shortform { -struct xfs_attr_sf_hdr {/* constant-structure header block */ -__be16 totsize;/* total bytes in shortform list */ -__u8 count;/* count of active entries */ -__u8 padding; - } hdr; -struct xfs_attr_sf_entry { -uint8_t namelen;/* actual length of name (no NULL) */ -uint8_t valuelen;/* actual length of value (no NULL) */ -uint8_t flags;/* flags bits (see xfs_attr_leaf.h) */ -uint8_t nameval[];/* name & value bytes concatenated */ - } list[];/* variable sized array */ +struct xfs_attr_sf_hdr {/* constant-structure header block */ +__be16 totsize;/* total bytes in shortform list */ +__u8 count;/* count of active entries */ +__u8 padding; +}; + +struct xfs_attr_sf_entry { +__u8 namelen;/* actual length of name (no NULL) */ +__u8 valuelen;/* actual length of value (no NULL) */ +__u8 flags;/* flags bits (XFS_ATTR_*) */ +__u8 nameval[];/* name & value bytes concatenated */ }; typedef struct xfs_attr_leaf_map { /* RLE map of free bytes */
libxfs/xfs_inode_fork.c +2 -3
@@ -277,10 +277,9 @@ static uint16_txfs_dfork_attr_shortform_size ( struct xfs_dinode *dip) { -struct xfs_attr_shortform *atp = - (struct xfs_attr_shortform *)XFS_DFORK_APTR (dip); +struct xfs_attr_sf_hdr *sf =XFS_DFORK_APTR (dip); -return be16_to_cpu (atp->hdr.totsize); +return be16_to_cpu (sf->totsize); }void
libxfs/xfs_ondisk.h +7 -7
@@ -93,13 +93,13 @@ xfs_check_ondisk_structs(void)XFS_CHECK_OFFSET (xfs_attr_leaf_name_remote_t, namelen,8 );XFS_CHECK_OFFSET (xfs_attr_leaf_name_remote_t, name,9 );XFS_CHECK_STRUCT_SIZE (xfs_attr_leafblock_t,32 ); - XFS_CHECK_STRUCT_SIZE(struct xfs_attr_shortform, 4); - XFS_CHECK_OFFSET(struct xfs_attr_shortform, hdr.totsize, 0); - XFS_CHECK_OFFSET(struct xfs_attr_shortform, hdr.count, 2); - XFS_CHECK_OFFSET(struct xfs_attr_shortform, list[0].namelen, 4); - XFS_CHECK_OFFSET(struct xfs_attr_shortform, list[0].valuelen, 5); - XFS_CHECK_OFFSET(struct xfs_attr_shortform, list[0].flags, 6); - XFS_CHECK_OFFSET(struct xfs_attr_shortform, list[0].nameval, 7); + XFS_CHECK_STRUCT_SIZE(struct xfs_attr_sf_hdr, 4); + XFS_CHECK_OFFSET(struct xfs_attr_sf_hdr, totsize, 0); + XFS_CHECK_OFFSET(struct xfs_attr_sf_hdr, count, 2); + XFS_CHECK_OFFSET(struct xfs_attr_sf_entry, namelen, 0); + XFS_CHECK_OFFSET(struct xfs_attr_sf_entry, valuelen, 1); + XFS_CHECK_OFFSET(struct xfs_attr_sf_entry, flags, 2); + XFS_CHECK_OFFSET(struct xfs_attr_sf_entry, nameval, 3);XFS_CHECK_STRUCT_SIZE (xfs_da_blkinfo_t,12 );XFS_CHECK_STRUCT_SIZE (xfs_da_intnode_t,16 );XFS_CHECK_STRUCT_SIZE (xfs_da_node_entry_t,8 );
repair/attr_repair.c +20 -24
116 lines not on this page · raw message
repair/dinode.c +11 -10
@@ -99,10 +99,10 @@ _("would have cleared inode %" PRIu64 " attributes\n"), ino_num);*/ if (!no_modify) { -struct xfs_attr_shortform *asf = (struct xfs_attr_shortform *) -XFS_DFORK_APTR (dino); - asf->hdr.totsize =cpu_to_be16 (sizeof (xfs_attr_sf_hdr_t)); - asf->hdr.count =0 ; +struct xfs_attr_sf_hdr *hdr =XFS_DFORK_APTR (dino); + + hdr->totsize =cpu_to_be16 (sizeof (struct xfs_attr_sf_hdr )); + hdr->count =0 ; dino->di_forkoff =0 ;/* got to do this after asf is set */ }
@@ -993,7 +993,7 @@ process_lclinode( struct xfs_dinode *dip, int whichfork) { -struct xfs_attr_shortform *asf; +struct xfs_attr_sf_hdr *hdr;xfs_ino_t lino; lino =XFS_AGINO_TO_INO (mp, agno, ino);
@@ -1005,18 +1005,19 @@ process_lclinode( XFS_DFORK_DSIZE(dip, mp));return (1 ); }else if (whichfork ==XFS_ATTR_FORK ) { - asf = (struct xfs_attr_shortform *)XFS_DFORK_APTR (dip); -if (be16_to_cpu (asf->hdr.totsize) >XFS_DFORK_ASIZE (dip, mp)) { + hdr =XFS_DFORK_APTR (dip); + +if (be16_to_cpu (hdr->totsize) >XFS_DFORK_ASIZE (dip, mp)) {do_warn (_ ("local inode %" PRIu64" attr fork too large (size %d, max = %zu)\n" ), - lino,be16_to_cpu (asf->hdr.totsize), + lino,be16_to_cpu (hdr->totsize),XFS_DFORK_ASIZE (dip, mp));return (1 ); } -if (be16_to_cpu (asf->hdr.totsize) <sizeof (xfs_attr_sf_hdr_t)) { +if (be16_to_cpu (hdr->totsize) <sizeof (xfs_attr_sf_hdr_t)) {do_warn (_ ("local inode %" PRIu64" attr too small (size = %d, min size = %zd)\n" ), - lino,be16_to_cpu (asf->hdr.totsize), + lino,be16_to_cpu (hdr->totsize),sizeof (xfs_attr_sf_hdr_t));return (1 ); }
PATCH 61/67 xfs: remove xfs_attr_sf_hdr_t
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 074aea4be1a4074be49a7ec41c674cc02b52fd60
Remove the last two users of the typedef.
--- db/attrshort.c | 2 +- libxfs/xfs_attr_leaf.c | 4 ++-- libxfs/xfs_attr_sf.h | 8 -------- repair/attr_repair.c | 6 +++--- repair/dinode.c | 4 ++-- 5 files changed, 8 insertions(+), 16 deletions(-)
db/attrshort.c +1 -1
@@ -25,7 +25,7 @@ const field_t attr_shortform_flds[] = { { NULL } }; -#define HOFF (f) bitize(offsetof(xfs_attr_sf_hdr_t, f)) +#define HOFF (f) bitize(offsetof(struct xfs_attr_sf_hdr, f))const field_t attr_sf_hdr_flds[] = { {"totsize" ,FLDT_UINT16D ,OI (HOFF (totsize)),C1 ,0 ,TYP_NONE }, {"count" ,FLDT_UINT8D ,OI (HOFF (count)),C1 ,0 ,TYP_NONE },
libxfs/xfs_attr_leaf.c +2 -2
@@ -813,7 +813,7 @@ xfs_attr_sf_removename(/* * Fix up the start offset of the attribute fork */ -if (totsize ==sizeof (xfs_attr_sf_hdr_t) &&xfs_has_attr2 (mp) && +if (totsize ==sizeof (struct xfs_attr_sf_hdr ) &&xfs_has_attr2 (mp) && (dp->i_df.if_format !=XFS_DINODE_FMT_BTREE ) && !(args->op_flags & (XFS_DA_OP_ADDNAME |XFS_DA_OP_REPLACE ))) {xfs_attr_fork_remove (dp, args->trans);
@@ -821,7 +821,7 @@ xfs_attr_sf_removename(xfs_idata_realloc (dp, -size,XFS_ATTR_FORK ); dp->i_forkoff =xfs_attr_shortform_bytesfit (dp, totsize);ASSERT (dp->i_forkoff); -ASSERT (totsize >sizeof (xfs_attr_sf_hdr_t) || +ASSERT (totsize >sizeof (struct xfs_attr_sf_hdr ) || (args->op_flags &XFS_DA_OP_ADDNAME ) || !xfs_has_attr2 (mp) || dp->i_df.if_format ==XFS_DINODE_FMT_BTREE );
libxfs/xfs_attr_sf.h +0 -8
@@ -6,14 +6,6 @@#ifndef __XFS_ATTR_SF_H__#define __XFS_ATTR_SF_H__ -/* -* Attribute storage when stored inside the inode. -* -* Small attribute lists are packed as tightly as possible so as -* to fit into the literal area of the inode. -*/ -typedef struct xfs_attr_sf_hdr xfs_attr_sf_hdr_t ; -/* * We generate this then sort it, attr_list() must return things in hash-order. */
repair/attr_repair.c +3 -3
@@ -222,14 +222,14 @@ process_shortform_attr(*/ if (hdr->count ==0 )/* then the total size should just be the header length */ -if (be16_to_cpu (hdr->totsize) !=sizeof (xfs_attr_sf_hdr_t)) { +if (be16_to_cpu (hdr->totsize) !=sizeof (struct xfs_attr_sf_hdr )) {/* whoops there's a discrepancy. Clear the hdr */ if (!no_modify) {do_warn (_ ("there are no attributes in the fork for inode %" PRIu64"\n" ), ino); hdr->totsize = -cpu_to_be16 (sizeof (xfs_attr_sf_hdr_t)); +cpu_to_be16 (sizeof (struct xfs_attr_sf_hdr )); *repair =1 ;return (1 ); }else {
@@ -240,7 +240,7 @@ process_shortform_attr( } } - currentsize =sizeof (xfs_attr_sf_hdr_t); + currentsize =sizeof (struct xfs_attr_sf_hdr ); remainingspace =be16_to_cpu (hdr->totsize) - currentsize; nextentry =libxfs_attr_sf_firstentry (hdr);for (i =0 ; i < hdr->count; i++) {
repair/dinode.c +2 -2
@@ -1014,11 +1014,11 @@ process_lclinode( XFS_DFORK_ASIZE(dip, mp));return (1 ); } -if (be16_to_cpu (hdr->totsize) <sizeof (xfs_attr_sf_hdr_t)) { +if (be16_to_cpu (hdr->totsize) <sizeof (struct xfs_attr_sf_hdr )) {do_warn (_ ("local inode %" PRIu64" attr too small (size = %d, min size = %zd)\n" ), lino,be16_to_cpu (hdr->totsize), -sizeof (xfs_attr_sf_hdr_t)); +sizeof (struct xfs_attr_sf_hdr ));return (1 ); } }
PATCH 62/67 xfs: turn the XFS_DA_OP_REPLACE checks in xfs_attr_shortform_addname into asserts
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 378b6aef9de0f7c3d0de309ecc61c11eb29e57da
Since commit deed9512872d ("xfs: Check for -ENOATTR or -EEXIST"), the high-level attr code does a lookup for any attr we're trying to set, and does the checks to handle the create vs replace cases, which thus never hit the low-level attr code.
Turn the checks in xfs_attr_shortform_addname as they must never trip.
--- libxfs/xfs_attr.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
libxfs/xfs_attr.c +2 -4
@@ -1070,8 +1070,7 @@ xfs_attr_shortform_addname(if (xfs_attr_sf_findname (args)) {int error; -if (!(args->op_flags &XFS_DA_OP_REPLACE )) -return -EEXIST ; +ASSERT (args->op_flags &XFS_DA_OP_REPLACE ); error =xfs_attr_sf_removename (args);if (error)
@@ -1085,8 +1084,7 @@ xfs_attr_shortform_addname(*/ args->op_flags &= ~XFS_DA_OP_REPLACE ; }else { -if (args->op_flags &XFS_DA_OP_REPLACE ) -return -ENOATTR ; +ASSERT (!(args->op_flags &XFS_DA_OP_REPLACE )); } if (args->namelen >= XFS_ATTR_SF_ENTSIZE_MAX ||
PATCH 63/67 xfs: fix a use after free in xfs_defer_finish_recovery
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: 4f6ac47b55e3ce6e982807928d6074ec105ab66e
dfp will be freed by ->recover_work and thus the tracepoint in case of an error can lead to a use after free.
Store the defer ops in a local variable to avoid that.
--- libxfs/xfs_defer.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
libxfs/xfs_defer.c +4 -2
@@ -909,12 +909,14 @@ xfs_defer_finish_recovery(struct xfs_defer_pending *dfp, struct list_head *capture_list) { + const struct xfs_defer_op_type *ops = dfp->dfp_ops;int error; - error = dfp->dfp_ops->recover_work (dfp, capture_list); +/* dfp is freed by recover_work and must not be accessed afterwards */ + error = ops->recover_work (dfp, capture_list);if (error)trace_xlog_intent_recovery_failed (mp, error, - dfp->dfp_ops->recover_work); + ops->recover_work);return error; }
PATCH 64/67 xfs: use the op name in trace_xlog_intent_recovery_failed
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Source kernel commit: bcdfae6ee520b665385020fa3e47633a8af84f12
Instead of tracing the address of the recovery handler, use the name in the defer op, similar to other defer ops related tracepoints.
--- libxfs/xfs_defer.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
libxfs/xfs_defer.c +1 -2
@@ -915,8 +915,7 @@ xfs_defer_finish_recovery(/* dfp is freed by recover_work and must not be accessed afterwards */ error = ops->recover_work (dfp, capture_list);if (error) -trace_xlog_intent_recovery_failed (mp, error, - ops->recover_work); +trace_xlog_intent_recovery_failed (mp, ops, error);return error; }
PATCH 65/67 xfs: fix backwards logic in xfs_bmap_alloc_account
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: d61b40bf15ce453f3aa71f6b423938e239e7f8f8
We're only allocating from the realtime device if the inode is marked for realtime and we're /not/ allocating into the attr fork.
--- libxfs/xfs_bmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
libxfs/xfs_bmap.c +1 -1
@@ -3271,7 +3271,7 @@ xfs_bmap_alloc_account( struct xfs_bmalloca *ap) {bool isrt =XFS_IS_REALTIME_INODE (ap->ip) && - (ap->flags &XFS_BMAPI_ATTRFORK ); + !(ap->flags &XFS_BMAPI_ATTRFORK );uint fld;if (ap->flags &XFS_BMAPI_COWFORK ) {
PATCH 66/67 xfs: reset XFS_ATTR_INCOMPLETE filter on node removal
In reply to Darrick J. WongFrom: Andrey Albershteyn <[email protected]>
Source kernel commit: 82ef1a5356572219f41f9123ca047259a77bd67b
In XFS_DAS_NODE_REMOVE_ATTR case, xfs_attr_mode_remove_attr() sets filter to XFS_ATTR_INCOMPLETE. The filter is then reset in xfs_attr_complete_op() if XFS_DA_OP_REPLACE operation is performed.
The filter is not reset though if XFS just removes the attribute (args->value == NULL) with xfs_attr_defer_remove(). attr code goes to XFS_DAS_DONE state.
Fix this by always resetting XFS_ATTR_INCOMPLETE filter. The replace operation already resets this filter in anyway and others are completed at this step hence don't need it.
--- libxfs/xfs_attr.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
libxfs/xfs_attr.c +3 -3
@@ -419,10 +419,10 @@ xfs_attr_complete_op(bool do_replace = args->op_flags &XFS_DA_OP_REPLACE ; args->op_flags &= ~XFS_DA_OP_REPLACE ; -if (do_replace) { - args->attr_filter &= ~XFS_ATTR_INCOMPLETE ; + args->attr_filter &= ~XFS_ATTR_INCOMPLETE ; +if (do_replace)return replace_state; - } +return XFS_DAS_DONE ; }
PATCH 67/67 xfs: remove conditional building of rt geometry validator functions
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Source kernel commit: 881f78f472556ed05588172d5b5676b48dc48240
I mistakenly turned off CONFIG_XFS_RT in the Kconfig file for arm64 variant of the djwong-wtf git branch. Unfortunately, it took me a good hour to figure out that RT wasn't built because this is what got printed to dmesg:
XFS (sda2): realtime geometry sanity check failed
XFS (sda2): Metadata corruption detected at xfs_sb_read_verify+0x170/0x190 [xfs], xfs_sb block 0x0
Whereas I would have expected:
XFS (sda2): Not built with CONFIG_XFS_RT
XFS (sda2): RT mount failed
The root cause of these problems is the conditional compilation of the new functions xfs_validate_rtextents and xfs_compute_rextslog that I introduced in the two commits listed below. The !RT versions of these functions return false and 0, respectively, which causes primary superblock validation to fail, which explains the first message.
Move the two functions to other parts of libxfs that are not conditionally defined by CONFIG_XFS_RT and remove the broken stubs so that validation works again.
--- libxfs/xfs_rtbitmap.c | 14 -------------- libxfs/xfs_rtbitmap.h | 16 ---------------- libxfs/xfs_sb.c | 14 ++++++++++++++ libxfs/xfs_sb.h | 2 ++ libxfs/xfs_types.h | 12 ++++++++++++ 5 files changed, 28 insertions(+), 30 deletions(-)
libxfs/xfs_rtbitmap.c +0 -14
@@ -1116,20 +1116,6 @@ xfs_rtbitmap_blockcount(return howmany_64 (rtextents,NBBY * mp->m_sb.sb_blocksize); } -/* -* Compute the maximum level number of the realtime summary file, as defined by -* mkfs. The historic use of highbit32 on a 64-bit quantity prohibited correct -* use of rt volumes with more than 2^32 extents. -*/ -uint8_t -xfs_compute_rextslog ( -xfs_rtbxlen_t rtextents) -{ -if (!rtextents) -return 0 ; -return xfs_highbit64 (rtextents); -} -/* * Compute the number of rtbitmap words needed to populate every block of a * bitmap that is large enough to track the given number of rt extents.
libxfs/xfs_rtbitmap.h +0 -16
@@ -351,20 +351,6 @@ xfs_rtfree_extent(int xfs_rtfree_blocks (struct xfs_trans *tp,xfs_fsblock_t rtbno,xfs_filblks_t rtlen); -uint8_t xfs_compute_rextslog (xfs_rtbxlen_t rtextents); - -/* Do we support an rt volume having this number of rtextents? */ -static inline bool -xfs_validate_rtextents ( -xfs_rtbxlen_t rtextents) -{ -/* No runt rt volumes */ -if (rtextents ==0 ) -return false; - -return true; -} -xfs_filblks_t xfs_rtbitmap_blockcount (struct xfs_mount *mp,xfs_rtbxlen_t rtextents); unsigned long long xfs_rtbitmap_wordcount(struct xfs_mount *mp,
@@ -383,8 +369,6 @@ unsigned long long xfs_rtsummary_wordcount(struct xfs_mount *mp,# define xfs_rtsummary_read_buf (a,b) (-ENOSYS)# define xfs_rtbuf_cache_relse (a) (0)# define xfs_rtalloc_extent_is_free (m,t,s,l,i) (-ENOSYS) -# define xfs_compute_rextslog (rtx) (0) -# define xfs_validate_rtextents (rtx) (false)static inline xfs_filblks_t xfs_rtbitmap_blockcount (struct xfs_mount *mp,xfs_rtbxlen_t rtextents) {
libxfs/xfs_sb.c +14 -0
@@ -1375,3 +1375,17 @@ xfs_validate_stripe_geometry( }return true; } + +/* +* Compute the maximum level number of the realtime summary file, as defined by +* mkfs. The historic use of highbit32 on a 64-bit quantity prohibited correct +* use of rt volumes with more than 2^32 extents. +*/ +uint8_t +xfs_compute_rextslog ( +xfs_rtbxlen_t rtextents) +{ +if (!rtextents) +return 0 ; +return xfs_highbit64 (rtextents); +}
libxfs/xfs_sb.h +2 -0
@@ -38,4 +38,6 @@ extern int xfs_sb_get_secondary(struct xfs_mount *mp,extern bool xfs_validate_stripe_geometry (struct xfs_mount *mp,__s64 sunit,__s64 swidth,int sectorsize,bool silent); +uint8_t xfs_compute_rextslog (xfs_rtbxlen_t rtextents); +#endif /* __XFS_SB_H__ */
libxfs/xfs_types.h +12 -0
@@ -251,4 +251,16 @@ bool xfs_verify_fileoff(struct xfs_mount *mp, xfs_fileoff_t off);bool xfs_verify_fileext (struct xfs_mount *mp,xfs_fileoff_t off,xfs_fileoff_t len); +/* Do we support an rt volume having this number of rtextents? */ +static inline bool +xfs_validate_rtextents ( +xfs_rtbxlen_t rtextents) +{ +/* No runt rt volumes */ +if (rtextents ==0 ) +return false; + +return true; +} +#endif /* __XFS_TYPES_H__ */
PATCH 3/18 xfs_repair: faster btree bulkloading
In reply to Darrick J. WongHi all,
Two improvements for xfs_repair: first, we adjust the btree bulk loader's slack computation to match the kernel. Second, we make the bulk loader write as many records as w can per ->get_records call.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-bulkload-faster
---
Commits in this patchset:
* xfs_repair: adjust btree bulkloading slack computations to match online repair
* xfs_repair: bulk load records into new btree blocks
--- repair/agbtree.c | 161 ++++++++++++++++++++++++++++++----------------------- repair/bulkload.c | 9 ++- 2 files changed, 95 insertions(+), 75 deletions(-)
PATCH 1/2 xfs_repair: adjust btree bulkloading slack computations to match online repair
From: Darrick J. Wong <[email protected]>
Adjust the lowspace threshold in the new btree block slack computation code to match online repair, which uses a straight 10% instead of magic shifting to approximate that without division. Repairs aren't that frequent in the kernel; and userspace can always do u64 division.
--- repair/bulkload.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
repair/bulkload.c +5 -4
@@ -106,9 +106,10 @@ bulkload_claim_block(* exceptions to this rule: * * (1) If someone turned one of the debug knobs. -* (2) The AG has less than ~9% space free. +* (2) The AG has less than ~10% space free. * -* Note that we actually use 3/32 for the comparison to avoid division. +* In the latter case, format the new btree blocks almost completely full to +* minimize space usage. */ void bulkload_estimate_ag_slack(
@@ -124,8 +125,8 @@ bulkload_estimate_ag_slack( bload->leaf_slack = bload_leaf_slack; bload->node_slack = bload_node_slack; -/* No further changes if there's more than 3/32ths space left. */ -if (free >= ((sc->mp->m_sb.sb_agblocks *3 ) >>5 )) +/* No further changes if there's more than 10% space left. */ +if (free >= sc->mp->m_sb.sb_agblocks /10 )return ;/*
PATCH 2/2 xfs_repair: bulk load records into new btree blocks
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Amortize the cost of indirect calls further by loading a batch of records into a new btree block instead of one record per ->get_record call. On a rmap btree with 3.9 million records, this reduces the runtime of xfs_btree_bload by 3% for xfsprogs. For the upcoming online repair functionality, this will reduce runtime by 6% when spectre mitigations are enabled in the kernel.
--- repair/agbtree.c | 161 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 90 insertions(+), 71 deletions(-)
repair/agbtree.c +90 -71
@@ -220,15 +220,19 @@ get_bnobt_records(struct bt_rebuild *btr = priv;struct xfs_alloc_rec_incore *arec = &cur->bc_rec.a;union xfs_btree_rec *block_rec; +unsigned int loaded; - btr->bno_rec =get_bno_rec (cur, btr->bno_rec); - arec->ar_startblock = btr->bno_rec->ex_startblock; - arec->ar_blockcount = btr->bno_rec->ex_blockcount; - btr->freeblks += btr->bno_rec->ex_blockcount; +for (loaded =0 ; loaded < nr_wanted; loaded++, idx++) { + btr->bno_rec =get_bno_rec (cur, btr->bno_rec); + arec->ar_startblock = btr->bno_rec->ex_startblock; + arec->ar_blockcount = btr->bno_rec->ex_blockcount; + btr->freeblks += btr->bno_rec->ex_blockcount; - block_rec =libxfs_btree_rec_addr (cur, idx, block); - cur->bc_ops->init_rec_from_cur (cur, block_rec); -return 1 ; + block_rec =libxfs_btree_rec_addr (cur, idx, block); + cur->bc_ops->init_rec_from_cur (cur, block_rec); + } + +return loaded; }void
@@ -388,65 +392,72 @@ get_inobt_records( {struct bt_rebuild *btr = priv;struct xfs_inobt_rec_incore *irec = &cur->bc_rec.i; -struct ino_tree_node *ino_rec; -union xfs_btree_rec *block_rec; -int inocnt =0 ; -int finocnt =0 ; -int k; - - btr->ino_rec = ino_rec =get_ino_rec (cur, btr->ino_rec); - -/* Transform the incore record into an on-disk record. */ - irec->ir_startino = ino_rec->ino_startnum; - irec->ir_free = ino_rec->ir_free; - -for (k =0 ; k <sizeof (xfs_inofree_t) *NBBY ; k++) { -ASSERT (is_inode_confirmed (ino_rec, k)); - -if (is_inode_sparse (ino_rec, k)) -continue ; -if (is_inode_free (ino_rec, k)) - finocnt++; - inocnt++; - } +unsigned int loaded =0 ; + +while (loaded < nr_wanted) { +struct ino_tree_node *ino_rec; +union xfs_btree_rec *block_rec; +int inocnt =0 ; +int finocnt =0 ; +int k; + + btr->ino_rec = ino_rec =get_ino_rec (cur, btr->ino_rec); - irec->ir_count = inocnt; - irec->ir_freecount = finocnt; - -if (xfs_has_sparseinodes (cur->bc_mp)) { -uint64_t sparse; -int spmask; -uint16_t holemask; - -/* -* Convert the 64-bit in-core sparse inode state to the -* 16-bit on-disk holemask. -*/ - holemask =0 ; - spmask = (1 <<XFS_INODES_PER_HOLEMASK_BIT ) -1 ; - sparse = ino_rec->ir_sparse; -for (k =0 ; k <XFS_INOBT_HOLEMASK_BITS ; k++) { -if (sparse & spmask) { -ASSERT ((sparse & spmask) == spmask); - holemask |= (1 << k); - }else -ASSERT ((sparse & spmask) ==0 ); - sparse >>=XFS_INODES_PER_HOLEMASK_BIT ; +/* Transform the incore record into an on-disk record. */ + irec->ir_startino = ino_rec->ino_startnum; + irec->ir_free = ino_rec->ir_free; + +for (k =0 ; k <sizeof (xfs_inofree_t) *NBBY ; k++) { +ASSERT (is_inode_confirmed (ino_rec, k)); + +if (is_inode_sparse (ino_rec, k)) +continue ; +if (is_inode_free (ino_rec, k)) + finocnt++; + inocnt++; } - irec->ir_holemask = holemask; - }else { - irec->ir_holemask =0 ; - } + irec->ir_count = inocnt; + irec->ir_freecount = finocnt; -if (btr->first_agino ==NULLAGINO ) - btr->first_agino = ino_rec->ino_startnum; - btr->freecount += finocnt; - btr->count += inocnt; +if (xfs_has_sparseinodes (cur->bc_mp)) { +uint64_t sparse; +int spmask; +uint16_t holemask; + +/* +* Convert the 64-bit in-core sparse inode state to the +* 16-bit on-disk holemask. +*/ + holemask =0 ; + spmask = (1 <<XFS_INODES_PER_HOLEMASK_BIT ) -1 ; + sparse = ino_rec->ir_sparse; +for (k =0 ; k <XFS_INOBT_HOLEMASK_BITS ; k++) { +if (sparse & spmask) { +ASSERT ((sparse & spmask) == spmask); + holemask |= (1 << k); + }else +ASSERT ((sparse & spmask) ==0 ); + sparse >>=XFS_INODES_PER_HOLEMASK_BIT ; + } + + irec->ir_holemask = holemask; + }else { + irec->ir_holemask =0 ; + } + +if (btr->first_agino ==NULLAGINO ) + btr->first_agino = ino_rec->ino_startnum; + btr->freecount += finocnt; + btr->count += inocnt; + + block_rec =libxfs_btree_rec_addr (cur, idx, block); + cur->bc_ops->init_rec_from_cur (cur, block_rec); + loaded++; + idx++; + } - block_rec =libxfs_btree_rec_addr (cur, idx, block); - cur->bc_ops->init_rec_from_cur (cur, block_rec); -return 1 ; +return loaded; }/* Initialize both inode btree cursors as needed. */
@@ -585,13 +596,17 @@ get_rmapbt_records(struct xfs_rmap_irec *rec;struct bt_rebuild *btr = priv;union xfs_btree_rec *block_rec; +unsigned int loaded; - rec =pop_slab_cursor (btr->slab_cursor); -memcpy (&cur->bc_rec.r, rec,sizeof (struct xfs_rmap_irec )); +for (loaded =0 ; loaded < nr_wanted; loaded++, idx++) { + rec =pop_slab_cursor (btr->slab_cursor); +memcpy (&cur->bc_rec.r, rec,sizeof (struct xfs_rmap_irec )); - block_rec =libxfs_btree_rec_addr (cur, idx, block); - cur->bc_ops->init_rec_from_cur (cur, block_rec); -return 1 ; + block_rec =libxfs_btree_rec_addr (cur, idx, block); + cur->bc_ops->init_rec_from_cur (cur, block_rec); + } + +return loaded; }/* Set up the rmap rebuild parameters. */
@@ -663,13 +678,17 @@ get_refcountbt_records(struct xfs_refcount_irec *rec;struct bt_rebuild *btr = priv;union xfs_btree_rec *block_rec; +unsigned int loaded; - rec =pop_slab_cursor (btr->slab_cursor); -memcpy (&cur->bc_rec.rc, rec,sizeof (struct xfs_refcount_irec )); +for (loaded =0 ; loaded < nr_wanted; loaded++, idx++) { + rec =pop_slab_cursor (btr->slab_cursor); +memcpy (&cur->bc_rec.rc, rec,sizeof (struct xfs_refcount_irec )); - block_rec =libxfs_btree_rec_addr (cur, idx, block); - cur->bc_ops->init_rec_from_cur (cur, block_rec); -return 1 ; + block_rec =libxfs_btree_rec_addr (cur, idx, block); + cur->bc_ops->init_rec_from_cur (cur, block_rec); + } + +return loaded; }/* Set up the refcount rebuild parameters. */
PATCH 4/18 xfsprogs: bug fixes for 6.8
In reply to Darrick J. WongHi all,
Bug fixes for xfsprogs for 6.8.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=xfsprogs-6.8-fixes
---
Commits in this patchset:
* xfs_repair: double-check with shortform attr verifiers
* xfs_db: fix alignment checks in getbitval
* xfs_scrub: fix threadcount estimates for phase 6
* xfs_scrub: don't fail while reporting media scan errors
* xfs_io: add linux madvise advice codes
--- db/bit.c | 9 +++--- io/madvise.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++- repair/attr_repair.c | 17 +++++++++++ scrub/phase6.c | 30 ++++++++++++++----- 4 files changed, 119 insertions(+), 14 deletions(-)
PATCH 1/5 xfs_repair: double-check with shortform attr verifiers
From: Darrick J. Wong <[email protected]>
Call the shortform attr structure verifier as the last thing we do in process_shortform_attr to make sure that we don't leave any latent errors for the kernel to stumble over.
--- repair/attr_repair.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
repair/attr_repair.c +17 -0
@@ -212,6 +212,7 @@ process_shortform_attr( {struct xfs_attr_sf_hdr *hdr =XFS_DFORK_APTR (dip);struct xfs_attr_sf_entry *currententry, *nextentry, *tempentry; +xfs_failaddr_t fa;int i, junkit;int currentsize, remainingspace;
@@ -373,6 +374,22 @@ process_shortform_attr( } } + fa =libxfs_attr_shortform_verify (hdr,be16_to_cpu (hdr->totsize)); +if (fa) { +if (no_modify) { +do_warn ( +_ ("inode %" PRIu64" shortform attr verifier failure, would have cleared attrs\n" ), + ino); + }else { +do_warn ( +_ ("inode %" PRIu64" shortform attr verifier failure, cleared attrs\n" ), + ino); + hdr->count =0 ; + hdr->totsize =cpu_to_be16 (sizeof (struct xfs_attr_sf_hdr )); + *repair =1 ; + } + } +return (*repair); }
PATCH 2/5 xfs_db: fix alignment checks in getbitval
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
For some reason, getbitval insists upon collecting a u64 from a pointer bit by bit if it's not aligned to a 16-byte boundary. Modern day systems only seem to require N-byte alignment for an N-byte quantity, so let's do that instead.
--- db/bit.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
db/bit.c +4 -5
@@ -62,24 +62,24 @@ getbitval( p = (char *)obj +byteize (bitoff); bit =bitoffs (bitoff); signext = (flags &BVSIGNED ) !=0 ; - z4 = ((intptr_t )p &0xf ) ==0 && bit ==0 ; + z4 = ((intptr_t )p & (sizeof (uint64_t ) -1 )) ==0 && bit ==0 ;if (nbits ==64 && z4)return be64_to_cpu (*(__be64 *)p); - z3 = ((intptr_t )p &0x7 ) ==0 && bit ==0 ; + z3 = ((intptr_t )p & (sizeof (uint32_t ) -1 )) ==0 && bit ==0 ;if (nbits ==32 && z3) {if (signext)return (__s32 )be32_to_cpu (*(__be32 *)p);else return (__u32 )be32_to_cpu (*(__be32 *)p); } - z2 = ((intptr_t )p &0x3 ) ==0 && bit ==0 ; + z2 = ((intptr_t )p & (sizeof (uint16_t ) -1 )) ==0 && bit ==0 ;if (nbits ==16 && z2) {if (signext)return (__s16 )be16_to_cpu (*(__be16 *)p);else return (__u16 )be16_to_cpu (*(__be16 *)p); } - z1 = ((intptr_t )p &0x1 ) ==0 && bit ==0 ; + z1 = bit ==0 ;if (nbits ==8 && z1) {if (signext)return *(__s8 *)p;
@@ -87,7 +87,6 @@ getbitval(return *(__u8 *)p; } -for (i =0 , rval =0LL ; i < nbits; i++) {if (getbit_l (p, bit + i)) {/* If the last bit is on and we care about sign
Re: [PATCH 2/5] xfs_db: fix alignment checks in getbitval
Quoted 6 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
For some reason, getbitval insists upon collecting a u64 from a pointer bit by bit if it's not aligned to a 16-byte boundary. Modern day systems only seem to require N-byte alignment for an N-byte quantity, so let's do that instead.
Not sure what modern day systems means here. In general in C you can do unaligned access, but it might be very inefficient.
If this code does what I think it does, maybe the right thing is to simply use the get_unaligned_be{16,32,64} helpers?
Re: [PATCH 2/5] xfs_db: fix alignment checks in getbitval
Quoted 10 lines · Christoph Hellwig
Quoted 6 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
For some reason, getbitval insists upon collecting a u64 from a pointer bit by bit if it's not aligned to a 16-byte boundary. Modern day systems only seem to require N-byte alignment for an N-byte quantity, so let's do that instead.
Not sure what modern day systems means here. In general in C you can do unaligned access, but it might be very inefficient.
Platforms like SPARC where unaligned accesses result in kernel traps that abort the program. Not that I've used any such platforms in 15 years now.
Quoted 2 lines
If this code does what I think it does, maybe the right thing is to simply use the get_unaligned_be{16,32,64} helpers?
Well we could still use the regular ones for aligned access, e.g.
#define PTR_ALIGNED(p, type) (((intptr_t)(p) & (sizeof(type) - 1)) == 0)
switch (nbits) {
case 64:
if (PTR_ALIGNED(p, __u64))
return be64_to_cpu(*(__be64 *)p);
return get_unaligned_be64(p);
...
}--D
Re: [PATCH 2/5] xfs_db: fix alignment checks in getbitval
Quoted 1 line · Darrick J. Wong
Well we could still use the regular ones for aligned access, e.g.
We could, but is it worth the effort? The few xfs_db command that do this bit en/decoding are ery much the definition of a slow path.
Re: [PATCH 2/5] xfs_db: fix alignment checks in getbitval
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Darrick J. Wong
Well we could still use the regular ones for aligned access, e.g.
We could, but is it worth the effort? The few
xfs_dbcommand that do this bit en/decoding are ery much the definition of a slow path.
Seeing as it's the debugger I don't really care about maximal performance anyway. :)
You might have noticed that the rtgroups patchset formally defines the rtbitmap words to have le32 ordering (instead of host ordering like we do now), at which point getbitval will need to be extended to handle little endian conversions. This is also useful for decoding fsverity descriptors.
So I'll go with the less clutter approach.
--D
PATCH 3/5 xfs_scrub: fix threadcount estimates for phase 6
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If a filesystem has a realtime device or an external log device, the media scan can start up a separate readverify controller (and workqueue) to handle that. Each of those controllers can call progress_add, so we need to bump up nr_threads so that the progress reports controller knows to make its ptvar big enough to handle all these threads.
--- scrub/phase6.c | 4 ++++ 1 file changed, 4 insertions(+)
scrub/phase6.c +4 -0
@@ -744,6 +744,10 @@ phase6_estimate( *items =cvt_off_fsb_to_b (&ctx->mnt, (d_blocks - d_bfree) + (r_blocks - r_bfree)); *nr_threads =disk_heads (ctx->datadev); +if (ctx->rtdev) + *nr_threads +=disk_heads (ctx->rtdev); +if (ctx->logdev) + *nr_threads +=disk_heads (ctx->logdev); *rshift =20 ;return 0 ; }
Re: [PATCH 3/5] xfs_scrub: fix threadcount estimates for phase 6
Quoted 7 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
If a filesystem has a realtime device or an external log device, the media scan can start up a separate readverify controller (and workqueue) to handle that. Each of those controllers can call
progress_add, so we need to bump upnr_threadsso that the progress reports controller knows to make its ptvar big enough to handle all these threads.
Maybe add a comment to the code stating this?
Re: [PATCH 3/5] xfs_scrub: fix threadcount estimates for phase 6
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
If a filesystem has a realtime device or an external log device, the media scan can start up a separate readverify controller (and workqueue) to handle that. Each of those controllers can call
progress_add, so we need to bump upnr_threadsso that the progress reports controller knows to make its ptvar big enough to handle all these threads.Maybe add a comment to the code stating this?
/* * Each read-verify pool starts a thread pool, and each worker thread * can contribute to the progress counter. Hence we need to set * nr_threads appropriately to handle that many threads. */ *nr_threads = disk_heads(ctx->datadev); if (ctx->rtdev) *nr_threads += disk_heads(ctx->rtdev); if (ctx->logdev) *nr_threads += disk_heads(ctx->logdev);
--D
Re: [PATCH 3/5] xfs_scrub: fix threadcount estimates for phase 6
Quoted 12 lines · Darrick J. Wong
Quoted 1 line
Maybe add a comment to the code stating this?
/* * Each read-verify pool starts a thread pool, and each worker thread * can contribute to the progress counter. Hence we need to set * nr_threads appropriately to handle that many threads. */ *nr_threads = disk_heads(ctx->datadev); if (ctx->rtdev) *nr_threads += disk_heads(ctx->rtdev); if (ctx->logdev) *nr_threads += disk_heads(ctx->logdev);
Looks reaѕonable.
PATCH 4/5 xfs_scrub: don't fail while reporting media scan errors
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If we can't open a file to report that it has media errors, just log that fact and move on. In this case we want to keep going with phase 6 so we report as many errors as possible.
--- scrub/phase6.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
scrub/phase6.c +18 -8
@@ -21,6 +21,7 @@#include "read_verify.h" #include "spacemap.h" #include "vfs.h" +#include "common.h" /* * Phase 6: Verify data file integrity.
@@ -291,13 +292,14 @@ report_inode_loss(/* Try to open the inode. */ fd =scrub_open_handle (handle);if (fd <0 ) { - error = errno; -if (error ==ESTALE ) -return error; +/* Handle is stale, try again. */ +if (errno ==ESTALE ) +return ESTALE ; -str_info (ctx, descr, -_ ("Disappeared during read error reporting." )); -return error; +str_error (ctx, descr, +_ ("Could not open to report read errors: %s." ), +strerror (errno)); +return 0 ; }/* Go find the badness. */
@@ -353,10 +355,18 @@ report_dirent_loss( fd =openat (dir_fd, dirent->d_name,O_RDONLY |O_NOATIME |O_NOFOLLOW |O_NOCTTY );if (fd <0 ) { +char descr[PATH_MAX +1 ]; +if (errno ==ENOENT )return 0 ; -str_errno (ctx, path); -return errno; + +snprintf (descr,PATH_MAX ,"%s/%s" , path, dirent->d_name); + descr[PATH_MAX ] =0 ; + +str_error (ctx, descr, +_ ("Could not open to report read errors: %s." ), +strerror (errno)); +return 0 ; }/* Go find the badness. */
Re: [PATCH 4/5] xfs_scrub: don't fail while reporting media scan errors
Looks good:
PATCH 5/5 xfs_io: add linux madvise advice codes
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add all the Linux-specific madvise codes. We're going to need MADV_POPULATE_READ for a regression test.
--- io/madvise.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 1 deletion(-)
io/madvise.c +76 -1
@@ -9,6 +9,7 @@#include <sys/mman.h> #include "init.h" #include "io.h" +#include <asm/mman.h> static cmdinfo_t madvise_cmd;
@@ -26,6 +27,31 @@ madvise_help(void) " -r -- expect random page references (POSIX_MADV_RANDOM)\n" " -s -- expect sequential page references (POSIX_MADV_SEQUENTIAL)\n" " -w -- will need these pages (POSIX_MADV_WILLNEED) [*]\n" +"\n" +"The following Linux-specific advise values are available:\n" +#ifdef MADV_COLLAPSE +" -c -- try to collapse range into transparent hugepages (MADV_COLLAPSE)\n" +#endif +#ifdef MADV_COLD +" -D -- deactivate the range (MADV_COLD)\n" +#endif +" -f -- free the range (MADV_FREE)\n" +" -h -- disable transparent hugepages (MADV_NOHUGEPAGE)\n" +" -H -- enable transparent hugepages (MADV_HUGEPAGE)\n" +" -m -- mark the range mergeable (MADV_MERGEABLE)\n" +" -M -- mark the range unmergeable (MADV_UNMERGEABLE)\n" +" -o -- mark the range offline (MADV_SOFT_OFFLINE)\n" +" -p -- punch a hole in the file (MADV_REMOVE)\n" +" -P -- poison the page cache (MADV_HWPOISON)\n" +#ifdef MADV_POPULATE_READ +" -R -- prefault in the range for read (MADV_POPULATE_READ)\n" +#endif +#ifdef MADV_POPULATE_WRITE +" -W -- prefault in the range for write (MADV_POPULATE_WRITE)\n" +#endif +#ifdef MADV_PAGEOUT +" -X -- reclaim the range (MADV_PAGEOUT)\n" +#endif " Notes:\n" " NORMAL sets the default readahead setting on the file.\n" " RANDOM sets the readahead setting on the file to zero.\n"
@@ -45,20 +71,69 @@ madvise_f(int advise =MADV_NORMAL , c;size_t blocksize, sectsize; -while ((c =getopt (argc, argv,"drsw" )) !=EOF ) { +while ((c =getopt (argc, argv,"cdDfhHmMopPrRswWX" )) !=EOF ) {switch (c) { +#ifdef MADV_COLLAPSE +case 'c' :/* collapse to thp */ + advise =MADV_COLLAPSE ; +break ; +#endif case 'd' :/* Don't need these pages */ advise =MADV_DONTNEED ;break ; +#ifdef MADV_COLD +case 'D' :/* make more likely to be reclaimed */ + advise =MADV_COLD ; +break ; +#endif +case 'f' :/* page range out of memory */ + advise =MADV_FREE ; +break ; +case 'h' :/* enable thp memory */ + advise =MADV_HUGEPAGE ; +break ; +case 'H' :/* disable thp memory */ + advise =MADV_NOHUGEPAGE ; +break ; +case 'm' :/* enable merging */ + advise =MADV_MERGEABLE ; +break ; +case 'M' :/* disable merging */ + advise =MADV_UNMERGEABLE ; +break ; +case 'o' :/* offline */ + advise =MADV_SOFT_OFFLINE ; +break ; +case 'p' :/* punch hole */ + advise =MADV_REMOVE ; +break ; +case 'P' :/* poison */ + advise =MADV_HWPOISON ; +break ;case 'r' :/* Expect random page references */ advise =MADV_RANDOM ;break ; +#ifdef MADV_POPULATE_READ +case 'R' :/* fault in pages for read */ + advise =MADV_POPULATE_READ ; +break ; +#endif case 's' :/* Expect sequential page references */ advise =MADV_SEQUENTIAL ;break ;case 'w' :/* Will need these pages */ advise =MADV_WILLNEED ;break ; +#ifdef MADV_POPULATE_WRITE +case 'W' :/* fault in pages for write */ + advise =MADV_POPULATE_WRITE ; +break ; +#endif +#ifdef MADV_PAGEOUT +case 'X' :/* reclaim memory */ + advise =MADV_PAGEOUT ; +break ; +#endif default : exitcode =1 ;return command_usage (&madvise_cmd);
Re: [PATCH 5/5] xfs_io: add linux madvise advice codes
Looks good:
PATCH v2 5/18 xfsprogs: fix log sector size detection
In reply to Darrick J. WongHi all,
From Christoph Hellwig,
this series cleans up the libxfs toplogy code and then fixes detection of the log sector size in mkfs.xfs, so that it doesn't create smaller than possible log sectors by default on > 512 byte sector size devices.
Note that this doesn't cleanup the types of the topology members, as that creeps all the way into platform_findsize. Which has a lot more cruft that should be dealth with and is worth it's own series.
Changes since v1:
- fix a spelling mistake
- add a few more cleanups
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=mkfs-fix-log-sector-size
---
Commits in this patchset:
* libxfs: remove the unused fs_topology_t typedef
* libxfs: refactor the fs_topology structure
* libxfs: remove the S_ISREG check from blkid_get_topology
* libxfs: also query log device topology in get_topology
* mkfs: use a sensible log sector size default
--- libxfs/topology.c | 109 ++++++++++++++++++++++++++--------------------------- libxfs/topology.h | 19 ++++++--- mkfs/xfs_mkfs.c | 71 ++++++++++++++++------------------- repair/sb.c | 2 - 4 files changed, 100 insertions(+), 101 deletions(-)
Re: [PATCHSET V2 05/18] xfsprogs: fix log sector size detection
Quoted 11 lines · Darrick J. Wong
Hi all,
From Christoph Hellwig,
this series cleans up the libxfs toplogy code and then fixes detection of the log sector size in mkfs.xfs, so that it doesn't create smaller than possible log sectors by default on > 512 byte sector size devices.
Note that this doesn't cleanup the types of the topology members, as that creeps all the way into
platform_findsize. Which has a lot more cruft that should be dealth with and is worth it's own series.
Oops, heh, in all the rebasing confusion I forgot to update this cover letter in my database. This is actually the V3 patchset, though I couldn't tell any difference between V2 and V3..
--D
Quoted 29 lines
Changes since v1:
- fix a spelling mistake
- add a few more cleanupsIf you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=mkfs-fix-log-sector-size
---
Commits in this patchset:
* libxfs: remove the unusedfs_topology_ttypedef
* libxfs: refactor thefs_topologystructure
* libxfs: remove theS_ISREGcheck fromblkid_get_topology
* libxfs: also query log device topology inget_topology
* mkfs: use a sensible log sector size default--- libxfs/topology.c | 109 ++++++++++++++++++++++++++--------------------------- libxfs/topology.h | 19 ++++++--- mkfs/xfs_mkfs.c | 71 ++++++++++++++++------------------- repair/sb.c | 2 - 4 files changed, 100 insertions(+), 101 deletions(-)
Re: [PATCHSET V2 05/18] xfsprogs: fix log sector size detection
Quoted 3 lines · Darrick J. Wong
Oops, heh, in all the rebasing confusion I forgot to update this cover letter in my database. This is actually the V3 patchset, though I couldn't tell any difference between V2 and V3..
The only difference was a tiny rebase fixup due to the configure cleanup series that went in.
PATCH 1/5 libxfs: remove the unused fs_topology_t typedef
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
--- libxfs/topology.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
libxfs/topology.h +2 -2
@@ -10,13 +10,13 @@/* * Device topology information. */ -typedef struct fs_topology { +struct fs_topology {int dsunit;/* stripe unit - data subvolume */ int dswidth;/* stripe width - data subvolume */ int rtswidth;/* stripe width - rt subvolume */ int lsectorsize;/* logical sector size &*/ int psectorsize;/* physical sector size */ -}fs_topology_t ; +};void get_topology(
PATCH 2/5 libxfs: refactor the fs_topology structure
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
fs_topology is a mess that mixes up data and RT device reporting, and to make things worse reuses lsectorsize for the logical sector size while other parts of xfsprogs use it for the log sector size.
Split out a device_topology structure that reports the topology for one device and embedded two of them into the fs_topology struture, and pass them directly to blkid_get_topology.
Rename the sector size members to be more explicit, and move some of the sanity checking from mkfs into the topology helpers.
--- libxfs/topology.c | 99 ++++++++++++++++++++++++++++------------------------- libxfs/topology.h | 14 +++++-- mkfs/xfs_mkfs.c | 64 ++++++++++++++++------------------ repair/sb.c | 2 + 4 files changed, 92 insertions(+), 87 deletions(-)
libxfs/topology.c +53 -46
@@ -170,18 +170,14 @@ check_overwrite(return ret; } -static void blkid_get_topology ( -const char *device, -int *sunit, -int *swidth, -int *lsectorsize, -int *psectorsize, -int force_overwrite) +static void +blkid_get_topology ( +const char *device, +struct device_topology *dt, +int force_overwrite) { -blkid_topology tp;blkid_probe pr; -unsigned long val;struct stat statbuf;/* can't get topology info from a file */
@@ -200,31 +196,28 @@ static void blkid_get_topology(if (!tp) goto out_free_probe; - val =blkid_topology_get_logical_sector_size (tp); - *lsectorsize = val; - val =blkid_topology_get_physical_sector_size (tp); - *psectorsize = val; - val =blkid_topology_get_minimum_io_size (tp); - *sunit = val; - val =blkid_topology_get_optimal_io_size (tp); - *swidth = val; + dt->logical_sector_size =blkid_topology_get_logical_sector_size (tp); + dt->physical_sector_size =blkid_topology_get_physical_sector_size (tp); + dt->sunit =blkid_topology_get_minimum_io_size (tp); + dt->swidth =blkid_topology_get_optimal_io_size (tp);/* * If the reported values are the same as the physical sector size * do not bother to report anything. It will only cause warnings * if people specify larger stripe units or widths manually. */ -if (*sunit == *psectorsize || *swidth == *psectorsize) { - *sunit =0 ; - *swidth =0 ; +if (dt->sunit == dt->physical_sector_size || + dt->swidth == dt->physical_sector_size) { + dt->sunit =0 ; + dt->swidth =0 ; }/* * Blkid reports the information in terms of bytes, but we want it in * terms of 512 bytes blocks (only to convert it to bytes later..) */ - *sunit = *sunit >>9 ; - *swidth = *swidth >>9 ; + dt->sunit >>=9 ; + dt->swidth >>=9 ;if (blkid_topology_get_alignment_offset (tp) !=0 ) { fprintf(stderr,
@@ -238,7 +231,7 @@ static void blkid_get_topology(exit (EXIT_FAILURE ); }/* Do not use physical sector size if the device is misaligned */ - *psectorsize = *lsectorsize; + dt->physical_sector_size = dt->logical_sector_size; }blkid_free_probe (pr);
@@ -251,47 +244,61 @@ static void blkid_get_topology( device); } -void -get_topology ( -struct libxfs_init *xi, -struct fs_topology *ft, +static void +get_device_topology ( +struct libxfs_dev *dev, +struct device_topology *dt,int force_overwrite) { -struct stat statbuf; +struct stat st; + +/* +* Nothing to do if this particular subvolume doesn't exist. +*/ +if (!dev->name) +return ;/* * If our target is a regular file, use platform_findsizes * to try to obtain the underlying filesystem's requirements * for direct IO; we'll set our sector size to that if possible. */ -if (xi->data.isfile || - (!stat (xi->data.name, &statbuf) &&S_ISREG (statbuf.st_mode))) { -int fd; +if (dev->isfile || (!stat (dev->name, &st) &&S_ISREG (st.st_mode))) {int flags =O_RDONLY ;long long dummy; +int fd;/* with xi->disfile we may not have the file yet! */ -if (xi->data.isfile) +if (dev->isfile) flags |=O_CREAT ; - fd =open (xi->data.name, flags,0666 ); + fd =open (dev->name, flags,0666 );if (fd >=0 ) { -platform_findsizes (xi->data.name, fd, &dummy, - &ft->lsectorsize); +platform_findsizes (dev->name, fd, &dummy, + &dt->logical_sector_size);close (fd); - ft->psectorsize = ft->lsectorsize; - }else - ft->psectorsize = ft->lsectorsize =BBSIZE ; + }else { + dt->logical_sector_size =BBSIZE ; + } }else { -blkid_get_topology (xi->data.name, &ft->dsunit, &ft->dswidth, - &ft->lsectorsize, &ft->psectorsize, - force_overwrite); +blkid_get_topology (dev->name, dt, force_overwrite); } -if (xi->rt.name && !xi->rt.isfile) { -int sunit, lsectorsize, psectorsize; +ASSERT (dt->logical_sector_size); -blkid_get_topology (xi->rt.name, &sunit, &ft->rtswidth, - &lsectorsize, &psectorsize, force_overwrite); - } +/* +* Older kernels may not have physical/logical distinction. +*/ +if (!dt->physical_sector_size) + dt->physical_sector_size = dt->logical_sector_size; +} + +void +get_topology ( +struct libxfs_init *xi, +struct fs_topology *ft, +int force_overwrite) +{ +get_device_topology (&xi->data, &ft->data, force_overwrite); +get_device_topology (&xi->rt, &ft->rt, force_overwrite); }
libxfs/topology.h +9 -5
@@ -10,12 +10,16 @@/* * Device topology information. */ +struct device_topology { +int logical_sector_size;/* logical sector size */ +int physical_sector_size;/* physical sector size */ +int sunit;/* stripe unit */ +int swidth;/* stripe width */ +}; +struct fs_topology { -int dsunit;/* stripe unit - data subvolume */ -int dswidth;/* stripe width - data subvolume */ -int rtswidth;/* stripe width - rt subvolume */ -int lsectorsize;/* logical sector size &*/ -int psectorsize;/* physical sector size */ +struct device_topology data; +struct device_topology rt; };void
mkfs/xfs_mkfs.c +29 -35
118 lines not on this page · raw message
repair/sb.c +1 -1
@@ -189,7 +189,7 @@ guess_default_geometry(* Use default block size (2^12) */ blocklog =12 ; - multidisk = ft.dswidth | ft.dsunit; + multidisk = ft.data.swidth | ft.data.sunit; dblocks = x->data.size >> (blocklog -BBSHIFT );calc_default_ag_geometry (blocklog, dblocks, multidisk, agsize, agcount);
PATCH 3/5 libxfs: remove the S_ISREG check from blkid_get_topology
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
The only caller already performs the exact same check.
--- libxfs/topology.c | 9 --------- 1 file changed, 9 deletions(-)
libxfs/topology.c +0 -9
@@ -178,15 +178,6 @@ blkid_get_topology( {blkid_topology tp;blkid_probe pr; -struct stat statbuf; - -/* can't get topology info from a file */ -if (!stat (device, &statbuf) &&S_ISREG (statbuf.st_mode)) { -fprintf (stderr, -_ ("%s: Warning: trying to probe topology of a file %s!\n" ), - progname, device); -return ; - } pr =blkid_new_probe_from_filename (device);if (!pr)
PATCH 4/5 libxfs: also query log device topology in get_topology
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Also query the log device topology in get_topology, which we'll need in mkfs in a bit.
--- libxfs/topology.c | 1 + libxfs/topology.h | 1 + 2 files changed, 2 insertions(+)
libxfs/topology.c +1 -0
@@ -292,4 +292,5 @@ get_topology( {get_device_topology (&xi->data, &ft->data, force_overwrite);get_device_topology (&xi->rt, &ft->rt, force_overwrite); +get_device_topology (&xi->log, &ft->log, force_overwrite); }
libxfs/topology.h +1 -0
@@ -20,6 +20,7 @@ struct device_topology {struct fs_topology {struct device_topology data;struct device_topology rt; +struct device_topology log; };void
PATCH 5/5 mkfs: use a sensible log sector size default
In reply to Darrick J. WongFrom: Christoph Hellwig <[email protected]>
Currently the XFS log sector size defaults to the 512 bytes unless explicitly overriden. Default to the device logical block size queried by get_topology instead. If that is also 512 nothing changes, and if the device logical block size is larger this prevents a mkfs failure because the libxfs buffer cache blows up and as we obviously can't use a smaller than hardware supported sector size. This fixes xfs/157 with a 4k block size device.
--- mkfs/xfs_mkfs.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
mkfs/xfs_mkfs.c +4 -3
@@ -2075,7 +2075,8 @@ static void validate_log_sectorsize( struct mkfs_params *cfg, struct cli_params *cli, - struct mkfs_default_params *dft) + struct mkfs_default_params *dft, + struct fs_topology *ft) { if (cli->loginternal && cli->lsectorsize &&
@@ -2090,7 +2091,7 @@ _("Can't change sector size on internal log!\n"));else if (cli->loginternal) cfg->lsectorsize = cfg->sectorsize;else - cfg->lsectorsize = dft->sectorsize; + cfg->lsectorsize = ft->log.logical_sector_size; cfg->lsectorlog =libxfs_highbit32 (cfg->lsectorsize); if (cfg->lsectorsize < XFS_MIN_SECTORSIZE ||
@@ -4206,7 +4207,7 @@ main( blocksize = cfg.blocksize; sectorsize = cfg.sectorsize; -validate_log_sectorsize (&cfg, &cli, &dft); +validate_log_sectorsize (&cfg, &cli, &dft, &ft);validate_sb_features (&cfg, &cli);/*
PATCH 6/18 mkfs: scale shards on ssds
In reply to Darrick J. WongHi all,
For a long time, the maintainers have had a gut feeling that we could optimize performance of XFS filesystems on non-mechanical storage by scaling the number of allocation groups to be a multiple of the CPU count.
With modern ~2022 hardware, it is common for systems to have more than four CPU cores and non-striped SSDs ranging in size from 256GB to 4TB. The default mkfs geometry still defaults to 4 AGs regardless of core count, which was settled on in the age of spinning rust.
This patchset adds a different computation for AG count and log size that is based entirely on a desired level of concurrency. If we detect storage that is non-rotational (or the sysadmin provides a CLI option), then we will try to match the AG count to the CPU count to minimize AGF contention and make the log large enough to minimize grant head contention.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=mkfs-scale-geo-on-ssds
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=mkfs-scale-geo-on-ssds
---
Commits in this patchset:
* mkfs: allow sizing allocation groups for concurrency
* mkfs: allow sizing internal logs for concurrency
--- man/man8/mkfs.xfs.8.in | 46 +++++++++ mkfs/xfs_mkfs.c | 251 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 291 insertions(+), 6 deletions(-)
PATCH 1/2 mkfs: allow sizing allocation groups for concurrency
From: Darrick J. Wong <[email protected]>
Add a -d concurrency= option to mkfs so that sysadmins can configure the filesystem so that there are enough allocation groups that the specified number of threads can (in theory) can find an uncontended group to allocate space from.
--- man/man8/mkfs.xfs.8.in | 27 +++++++++ mkfs/xfs_mkfs.c | 150 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 173 insertions(+), 4 deletions(-)
man/man8/mkfs.xfs.8.in +27 -0
@@ -504,6 +504,33 @@ directories. By default, .B mkfs.xfs will not enable DAX mode. +.TP +.BI concurrency= value +Create enough allocation groups to handle the desired level of concurrency. +The goal of this calculation scheme is to set the number of allocation groups +to an integer multiple of the number of writer threads desired, to minimize +contention of AG locks. +This scheme will neither create fewer AGs than would be created by the default +configuration, nor will it create AGs smaller than 4GB. +This option is not compatible with the +.B agcount +or +.B agsize +options. +The magic value +.I nr_cpus +or +.I 1 +or no value at all will set this parameter to the number of active processors +in the system. +If the kernel advertises that the data device is a non-mechanical storage +device, +.B mkfs.xfs +will use this new geometry calculation scheme. +The magic value of +.I 0 +forces use of the older AG geometry calculations that is used for mechanical +storage. .RE .TP .B \-f
mkfs/xfs_mkfs.c +146 -4
237 lines not on this page · raw message
PATCH 2/2 mkfs: allow sizing internal logs for concurrency
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a -l option to mkfs so that sysadmins can configure the filesystem so that the log can handle a certain number of transactions (front and backend) without any threads contending for log grant space.
--- man/man8/mkfs.xfs.8.in | 19 +++++++++ mkfs/xfs_mkfs.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 118 insertions(+), 2 deletions(-)
man/man8/mkfs.xfs.8.in +19 -0
@@ -795,6 +795,25 @@ if you want to disable this feature for older kernels which don't support it. .IP This option is only tunable on the deprecated V4 format. +.TP +.BI concurrency= value +Allocate a log that is estimated to be large enough to handle the desired level +of concurrency without userspace program threads contending for log space. +This scheme will neither create a log smaller than the minimum required, +nor create a log larger than the maximum possible. +This option is only valid for internal logs and is not compatible with the +size option. +This option is not compatible with the +.B logdev +or +.B size +options. +The magic value +.I nr_cpus +or +.I 1 +or no value at all will set this parameter to the number of active processors +in the system. .RE .PP .PD 0
mkfs/xfs_mkfs.c +99 -2
191 lines not on this page · raw message
PATCH 7/18 xfs_scrub: scan metadata files in parallel
In reply to Darrick J. WongHi all,
At this point, we need to clean up the libfrog and xfs_scrub code a little bit. First, correct some of the weird naming and organizing choices I made in libfrog for scrub types and fs summary counter scans. Second, break out metadata file scans as a separate group, and teach xfs_scrub that it can ask the kernel to scan them in parallel. On filesystems with quota or realtime volumes, this can speed up that part significantly.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-metafile-parallel
---
Commits in this patchset:
* libfrog: rename XFROG_SCRUB_TYPE_* to XFROG_SCRUB_GROUP_*
* libfrog: promote XFROG_SCRUB_DESCR_SUMMARY to a scrub type
* xfs_scrub: scan whole-fs metadata files in parallel
--- io/scrub.c | 13 +++-- libfrog/scrub.c | 51 ++++++++++----------- libfrog/scrub.h | 24 ++++------ scrub/phase2.c | 135 ++++++++++++++++++++++++++++++++++++++++++------------- scrub/phase4.c | 2 - scrub/phase7.c | 4 +- scrub/scrub.c | 75 ++++++++++++++++++------------- scrub/scrub.h | 6 ++ 8 files changed, 194 insertions(+), 116 deletions(-)
PATCH 1/3 libfrog: rename XFROG_SCRUB_TYPE_* to XFROG_SCRUB_GROUP_*
From: Darrick J. Wong <[email protected]>
I didn't do a good job of naming XFROG_SCRUB_TYPE when I created that enumeration. The goal of the enum is to group the scrub ioctl's XFS_SCRUB_TYPE_* codes by principal filesystem object (AG, inode, etc.) but for some dumb reason I chose to reuse "type". This is confusing, so fix this sin.
--- io/scrub.c | 12 ++++++------ libfrog/scrub.c | 50 +++++++++++++++++++++++++------------------------- libfrog/scrub.h | 16 ++++++++-------- scrub/scrub.c | 54 +++++++++++++++++++++++++++--------------------------- 4 files changed, 66 insertions(+), 66 deletions(-)
io/scrub.c +6 -6
@@ -166,23 +166,23 @@ parse_args( meta->sm_type = type; meta->sm_flags = flags; -switch (d->type) { -case XFROG_SCRUB_TYPE_INODE : +switch (d->group) { +case XFROG_SCRUB_GROUP_INODE :if (!parse_inode (argc, argv, optind, &meta->sm_ino, &meta->sm_gen)) { exitcode =1 ;return command_usage (cmdinfo); }break ; -case XFROG_SCRUB_TYPE_AGHEADER : -case XFROG_SCRUB_TYPE_PERAG : +case XFROG_SCRUB_GROUP_AGHEADER : +case XFROG_SCRUB_GROUP_PERAG :if (!parse_agno (argc, argv, optind, &meta->sm_agno)) { exitcode =1 ;return command_usage (cmdinfo); }break ; -case XFROG_SCRUB_TYPE_FS : -case XFROG_SCRUB_TYPE_NONE : +case XFROG_SCRUB_GROUP_FS : +case XFROG_SCRUB_GROUP_NONE :if (!parse_none (argc, optind)) { exitcode =1 ;return command_usage (cmdinfo);
libfrog/scrub.c +25 -25
152 lines not on this page · raw message
libfrog/scrub.h +8 -8
@@ -6,20 +6,20 @@#ifndef __LIBFROG_SCRUB_H__#define __LIBFROG_SCRUB_H__ -/* Type info and names for the scrub types. */ -enum xfrog_scrub_type { -XFROG_SCRUB_TYPE_NONE ,/* not metadata */ -XFROG_SCRUB_TYPE_AGHEADER ,/* per-AG header */ -XFROG_SCRUB_TYPE_PERAG ,/* per-AG metadata */ -XFROG_SCRUB_TYPE_FS ,/* per-FS metadata */ -XFROG_SCRUB_TYPE_INODE ,/* per-inode metadata */ +/* Group the scrub types by principal filesystem object. */ +enum xfrog_scrub_group { +XFROG_SCRUB_GROUP_NONE ,/* not metadata */ +XFROG_SCRUB_GROUP_AGHEADER ,/* per-AG header */ +XFROG_SCRUB_GROUP_PERAG ,/* per-AG metadata */ +XFROG_SCRUB_GROUP_FS ,/* per-FS metadata */ +XFROG_SCRUB_GROUP_INODE ,/* per-inode metadata */ };/* Catalog of scrub types and names, indexed by XFS_SCRUB_TYPE_* */ struct xfrog_scrub_descr {const char *name;const char *descr; -enum xfrog_scrub_type type; +enum xfrog_scrub_group group;unsigned int flags; };
scrub/scrub.c +27 -27
133 lines not on this page · raw message
PATCH 2/3 libfrog: promote XFROG_SCRUB_DESCR_SUMMARY to a scrub type
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
"Summary" metadata, at least in the scrub context, are metadata whose values depend on some kind of computation and therefore can only be checked after we've looked at all the other metadata. Currently, the superblock summary counters are the only thing that are like this, but since they run in a totally separate xfs_scrub phase (7 vs. 2), make them their own group and remove the group+flag mix.
--- io/scrub.c | 1 + libfrog/scrub.c | 3 +-- libfrog/scrub.h | 8 +------- scrub/phase4.c | 2 +- scrub/phase7.c | 4 ++-- scrub/scrub.c | 16 ++++++++++++---- scrub/scrub.h | 3 ++- 7 files changed, 20 insertions(+), 17 deletions(-)
io/scrub.c +1 -0
@@ -183,6 +183,7 @@ parse_args(break ;case XFROG_SCRUB_GROUP_FS :case XFROG_SCRUB_GROUP_NONE : +case XFROG_SCRUB_GROUP_SUMMARY :if (!parse_none (argc, optind)) { exitcode =1 ;return command_usage (cmdinfo);
libfrog/scrub.c +1 -2
@@ -132,8 +132,7 @@ const struct xfrog_scrub_descr xfrog_scrubbers[XFS_SCRUB_TYPE_NR] = { [XFS_SCRUB_TYPE_FSCOUNTERS ] = { .name ="fscounters" , .descr ="filesystem summary counters" , - .group =XFROG_SCRUB_GROUP_FS , - .flags =XFROG_SCRUB_DESCR_SUMMARY , + .group =XFROG_SCRUB_GROUP_SUMMARY , }, };
libfrog/scrub.h +1 -7
@@ -13,6 +13,7 @@ enum xfrog_scrub_group {XFROG_SCRUB_GROUP_PERAG ,/* per-AG metadata */ XFROG_SCRUB_GROUP_FS ,/* per-FS metadata */ XFROG_SCRUB_GROUP_INODE ,/* per-inode metadata */ + XFROG_SCRUB_GROUP_SUMMARY, /* summary metadata */ };/* Catalog of scrub types and names, indexed by XFS_SCRUB_TYPE_* */
@@ -20,15 +21,8 @@ struct xfrog_scrub_descr {const char *name;const char *descr;enum xfrog_scrub_group group; -unsigned int flags; }; -/* -* The type of metadata checked by this scrubber is a summary of other types -* of metadata. This scrubber should be run after all the others. -*/ -#define XFROG_SCRUB_DESCR_SUMMARY (1 << 0) -extern const struct xfrog_scrub_descr xfrog_scrubbers[XFS_SCRUB_TYPE_NR ];int xfrog_scrub_metadata (struct xfs_fd *xfd,struct xfs_scrub_metadata *meta);
scrub/phase4.c +1 -1
@@ -139,7 +139,7 @@ phase4_func(* counters, so counter repairs have to be put on the list now so that * they get fixed before we stop retrying unfixed metadata repairs. */ - ret =scrub_fs_summary (ctx, &ctx->action_lists[0 ]); + ret =scrub_fs_counters (ctx, &ctx->action_lists[0 ]);if (ret)return ret;
scrub/phase7.c +2 -2
@@ -116,9 +116,9 @@ phase7_func(int ip;int error; -/* Check and fix the fs summary counters. */ +/* Check and fix the summary metadata. */ action_list_init (&alist); - error =scrub_fs_summary (ctx, &alist); + error =scrub_summary_metadata (ctx, &alist);if (error)return error; error = action_list_process(ctx, -1, &alist,
scrub/scrub.c +12 -4
@@ -46,6 +46,7 @@ format_scrub_descr( _(sc->descr));break ;case XFROG_SCRUB_GROUP_FS : +case XFROG_SCRUB_GROUP_SUMMARY :return snprintf (buf, buflen,_ ("%s" ),_ (sc->descr));break ;case XFROG_SCRUB_GROUP_NONE :
@@ -356,8 +357,6 @@ scrub_group(if (sc->group != group)continue ; -if (sc->flags &XFROG_SCRUB_DESCR_SUMMARY ) -continue ; ret =scrub_meta_type (ctx, type, agno, alist);if (ret)
@@ -410,9 +409,18 @@ scrub_fs_metadata(return scrub_group (ctx,XFROG_SCRUB_GROUP_FS ,0 , alist); } -/* Scrub FS summary metadata. */ +/* Scrub all FS summary metadata. */ int -scrub_fs_summary ( +scrub_summary_metadata ( +struct scrub_ctx *ctx, +struct action_list *alist) +{ +return scrub_group (ctx,XFROG_SCRUB_GROUP_SUMMARY ,0 , alist); +} + +/* Scrub /only/ the superblock summary counters. */ +int +scrub_fs_counters (struct scrub_ctx *ctx,struct action_list *alist) {
scrub/scrub.h +2 -1
@@ -23,7 +23,8 @@ int scrub_ag_headers(struct scrub_ctx *ctx, xfs_agnumber_t agno,int scrub_ag_metadata (struct scrub_ctx *ctx,xfs_agnumber_t agno,struct action_list *alist);int scrub_fs_metadata (struct scrub_ctx *ctx,struct action_list *alist); -int scrub_fs_summary (struct scrub_ctx *ctx,struct action_list *alist); +int scrub_summary_metadata (struct scrub_ctx *ctx,struct action_list *alist); +int scrub_fs_counters (struct scrub_ctx *ctx,struct action_list *alist);bool can_scrub_fs_metadata (struct scrub_ctx *ctx);bool can_scrub_inode (struct scrub_ctx *ctx);
PATCH 3/3 xfs_scrub: scan whole-fs metadata files in parallel
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
The realtime bitmap and the three quota files are completely independent of each other, which means that we ought to be able to scan them in parallel. Rework the phase2 code so that we can do this. Note, however, that the realtime summary file summarizes the contents of the realtime bitmap, so we must coordinate the workqueue threads.
--- scrub/phase2.c | 135 +++++++++++++++++++++++++++++++++++++++++++------------- scrub/scrub.c | 7 ++- scrub/scrub.h | 3 + 3 files changed, 110 insertions(+), 35 deletions(-)
scrub/phase2.c +103 -32
@@ -10,6 +10,8 @@#include "list.h" #include "libfrog/paths.h" #include "libfrog/workqueue.h" +#include "libfrog/fsgeom.h" +#include "libfrog/scrub.h" #include "xfs_scrub.h" #include "common.h" #include "scrub.h"
@@ -17,6 +19,18 @@/* Phase 2: Check internal metadata. */ +struct scan_ctl { +/* +* Control mechanism to signal that the rt bitmap file scan is done and +* wake up any waiters. +*/ +pthread_cond_t rbm_wait; +pthread_mutex_t rbm_waitlock; +bool rbm_done; + +bool aborted; +}; +/* Scrub each AG's metadata btrees. */ static void scan_ag_metadata(
@@ -25,7 +39,7 @@ scan_ag_metadata( void *arg) {struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx; -bool *aborted = arg; +struct scan_ctl *sctl = arg;struct action_list alist;struct action_list immediate_alist;unsigned long long broken_primaries;
@@ -33,7 +47,7 @@ scan_ag_metadata(char descr[DESCR_BUFSZ ];int ret; -if (*aborted) +if (sctl->aborted)return ;action_list_init (&alist);
@@ -89,32 +103,40 @@ _("Filesystem might not be repairable."));action_list_defer (ctx, agno, &alist);return ; err: - *aborted = true; + sctl->aborted = true; } -/* Scrub whole-FS metadata btrees. */ +/* Scan whole-fs metadata. */ static void scan_fs_metadata ( -struct workqueue *wq, -xfs_agnumber_t agno, -void *arg) +struct workqueue *wq, +xfs_agnumber_t type, +void *arg) { -struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx; -bool *aborted = arg; -struct action_list alist; -int ret; +struct action_list alist; +struct scrub_ctx *ctx = (struct scrub_ctx *)wq->wq_ctx; +struct scan_ctl *sctl = arg; +int ret; -if (*aborted) -return ; +if (sctl->aborted) + goto out;action_list_init (&alist); - ret =scrub_fs_metadata (ctx, &alist); + ret =scrub_fs_metadata (ctx, type, &alist);if (ret) { - *aborted = true; -return ; + sctl->aborted = true; + goto out; } -action_list_defer (ctx, agno, &alist); +action_list_defer (ctx,0 , &alist); + +out: +if (type ==XFS_SCRUB_TYPE_RTBITMAP ) { +pthread_mutex_lock (&sctl->rbm_waitlock); + sctl->rbm_done = true; +pthread_cond_broadcast (&sctl->rbm_wait); +pthread_mutex_unlock (&sctl->rbm_waitlock); + } }/* Scan all filesystem metadata. */
@@ -122,17 +144,25 @@ intphase2_func ( struct scrub_ctx *ctx) { -struct action_list alist;struct workqueue wq; +struct scan_ctl sctl = { + .aborted = false, + .rbm_done = false, + }; +struct action_list alist; +const struct xfrog_scrub_descr *sc = xfrog_scrubbers;xfs_agnumber_t agno; -bool aborted = false; +unsigned int type;int ret, ret2; +pthread_mutex_init (&sctl.rbm_waitlock,NULL ); +pthread_cond_init (&sctl.rbm_wait,NULL ); + ret = -workqueue_create (&wq, (struct xfs_mount *)ctx,scrub_nproc_workqueue (ctx));if (ret) {str_liberror (ctx, ret,_ ("creating scrub workqueue" )); -return ret; + goto out_wait; }/*
@@ -143,29 +173,67 @@ phase2_func(action_list_init (&alist); ret =scrub_primary_super (ctx, &alist);if (ret) - goto out; + goto out_wq; ret =action_list_process_or_defer (ctx,0 , &alist);if (ret) - goto out; + goto out_wq; -for (agno =0 ; !aborted && agno < ctx->mnt.fsgeom.agcount; agno++) { - ret = -workqueue_add (&wq, scan_ag_metadata, agno, &aborted); +/* Scan each AG in parallel. */ +for (agno =0 ; + agno < ctx->mnt.fsgeom.agcount && !sctl.aborted; + agno++) { + ret = -workqueue_add (&wq, scan_ag_metadata, agno, &sctl);if (ret) {str_liberror (ctx, ret,_ ("queueing per-AG scrub work" )); - goto out; + goto out_wq; } } -if (aborted) - goto out; +if (sctl.aborted) + goto out_wq; - ret = -workqueue_add (&wq, scan_fs_metadata,0 , &aborted); +/* +* Scan all of the whole-fs metadata objects: realtime bitmap, realtime +* summary, and the three quota files. Each of the metadata files can +* be scanned in parallel except for the realtime summary file, which +* must run after the realtime bitmap has been scanned. +*/ +for (type =0 ; type <XFS_SCRUB_TYPE_NR ; type++, sc++) { +if (sc->group !=XFROG_SCRUB_GROUP_FS ) +continue ; +if (type ==XFS_SCRUB_TYPE_RTSUM ) +continue ; + + ret = -workqueue_add (&wq, scan_fs_metadata, type, &sctl); +if (ret) { +str_liberror (ctx, ret, +_ ("queueing whole-fs scrub work" )); + goto out_wq; + } + } + +if (sctl.aborted) + goto out_wq; + +/* +* Wait for the rt bitmap to finish scanning, then scan the rt summary +* since the summary can be regenerated completely from the bitmap. +*/ +pthread_mutex_lock (&sctl.rbm_waitlock); +while (!sctl.rbm_done) +pthread_cond_wait (&sctl.rbm_wait, &sctl.rbm_waitlock); +pthread_mutex_unlock (&sctl.rbm_waitlock); + +if (sctl.aborted) + goto out_wq; + + ret = -workqueue_add (&wq, scan_fs_metadata,XFS_SCRUB_TYPE_RTSUM , &sctl);if (ret) { -str_liberror (ctx, ret,_ ("queueing per-FS scrub work" )); - goto out; +str_liberror (ctx, ret,_ ("queueing rtsummary scrub work" )); + goto out_wq; } -out: +out_wq: ret2 = -workqueue_terminate (&wq);if (ret2) {str_liberror (ctx, ret2,_ ("finishing scrub work" ));
@@ -173,8 +241,11 @@ phase2_func( ret = ret2; }workqueue_destroy (&wq); +out_wait: +pthread_cond_destroy (&sctl.rbm_wait); +pthread_mutex_destroy (&sctl.rbm_waitlock); -if (!ret && aborted) +if (!ret && sctl.aborted) ret =ECANCELED ;return ret; }
scrub/scrub.c +5 -2
@@ -400,13 +400,16 @@ scrub_ag_metadata(return scrub_group (ctx,XFROG_SCRUB_GROUP_PERAG , agno, alist); } -/* Scrub whole-FS metadata btrees. */ +/* Scrub whole-filesystem metadata. */ int scrub_fs_metadata (struct scrub_ctx *ctx, +unsigned int type,struct action_list *alist) { -return scrub_group (ctx,XFROG_SCRUB_GROUP_FS ,0 , alist); +ASSERT (xfrog_scrubbers[type].group ==XFROG_SCRUB_GROUP_FS ); + +return scrub_meta_type (ctx, type,0 , alist); }/* Scrub all FS summary metadata. */
scrub/scrub.h +2 -1
@@ -22,7 +22,8 @@ int scrub_ag_headers(struct scrub_ctx *ctx, xfs_agnumber_t agno, struct action_list *alist);int scrub_ag_metadata (struct scrub_ctx *ctx,xfs_agnumber_t agno,struct action_list *alist); -int scrub_fs_metadata (struct scrub_ctx *ctx,struct action_list *alist); +int scrub_fs_metadata (struct scrub_ctx *ctx,unsigned int scrub_type, +struct action_list *alist);int scrub_summary_metadata (struct scrub_ctx *ctx,struct action_list *alist);int scrub_fs_counters (struct scrub_ctx *ctx,struct action_list *alist);
PATCH 8/18 xfs_repair: rebuild inode fork mappings
In reply to Darrick J. WongHi all,
Add the ability to regenerate inode fork mappings if the rmapbt otherwise looks ok.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-rebuild-forks
---
Commits in this patchset:
* xfs_repair: push inode buf and dinode pointers all the way to inode fork processing
* xfs_repair: sync bulkload data structures with kernel newbt code
* xfs_repair: rebuild block mappings from rmapbt data
--- include/xfs_trans.h | 2 libfrog/util.h | 5 libxfs/libxfs_api_defs.h | 16 + libxfs/trans.c | 48 +++ repair/Makefile | 2 repair/agbtree.c | 24 + repair/bmap_repair.c | 748 ++++++++++++++++++++++++++++++++++++++++++++++ repair/bmap_repair.h | 13 + repair/bulkload.c | 260 +++++++++++++++- repair/bulkload.h | 34 ++ repair/dino_chunks.c | 5 repair/dinode.c | 142 ++++++--- repair/dinode.h | 7 repair/phase5.c | 2 repair/rmap.c | 2 repair/rmap.h | 1 16 files changed, 1231 insertions(+), 80 deletions(-) create mode 100644 repair/bmap_repair.c create mode 100644 repair/bmap_repair.h
PATCH 1/3 xfs_repair: push inode buf and dinode pointers all the way to inode fork processing
From: Darrick J. Wong <[email protected]>
Currently, the process_dinode* family of functions assume that they have the buffer backing the inodes locked, and therefore the dinode pointer won't ever change. However, the bmbt rebuilding code in the next patch will violate that assumption, so we must pass pointers to the inobp and the dinode pointer (that is to say, double pointers) all the way through to process_inode_{data,attr}_fork so that we can regrab the buffer after the rebuilding step finishes.
--- repair/dino_chunks.c | 5 ++- repair/dinode.c | 88 ++++++++++++++++++++++++++++---------------------- repair/dinode.h | 7 ++-- 3 files changed, 57 insertions(+), 43 deletions(-)
repair/dino_chunks.c +3 -2
@@ -851,10 +851,11 @@ process_inode_chunk( ino_dirty =0 ; parent =0 ; - status =process_dinode (mp, dino, agno, agino, + status =process_dinode (mp, &dino, agno, agino,is_inode_free (ino_rec, irec_offset), &ino_dirty, &is_used,ino_discovery, check_dups, - extra_attr_check, &isa_dir, &parent); + extra_attr_check, &isa_dir, &parent, + &bplist[bp_index]);ASSERT (is_used !=3 );if (ino_dirty) {
repair/dinode.c +50 -38
185 lines not on this page · raw message
repair/dinode.h +4 -3
@@ -43,8 +43,8 @@ voidupdate_rootino (xfs_mount_t *mp); int -process_dinode(xfs_mount_t *mp, - struct xfs_dinode *dino, +process_dinode(struct xfs_mount *mp, + struct xfs_dinode **dinop, xfs_agnumber_t agno, xfs_agino_t ino, int was_free,
@@ -54,7 +54,8 @@ process_dinode(xfs_mount_t *mp,int check_dups, int extra_attr_check, int *isa_dir, - xfs_ino_t *parent); + xfs_ino_t *parent, + struct xfs_buf **ino_bpp); int verify_dinode(xfs_mount_t *mp,
PATCH 2/3 xfs_repair: sync bulkload data structures with kernel newbt code
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
A lot of the code in repair/bulkload.c was backwardsported from new code that eventually turned into newbt.c in online repair. Since the offline repair version got merged upstream years before the online repair code, we now need to bring the offline version up to date with the kernel again.
Right now, the bulkload.c code is just a fancy way to track space extents that are fed to it by its callers. The only caller, of course, is phase 5, which builds new btrees in AG space that wasn't claimed by any other data structure. Hence there's no need to allocate reservations out of the bnobt or put them back there.
However, the next patch adds the ability to generate new file-based btrees. For that we need to reorganize the code to allocate and free space for new file-based btrees. Let's just crib from the kernel version. Make each bulkload space reservation hold a reference to an AG and track the space reservation in terms of per-AG extents instead of fsblock extents.
--- libxfs/libxfs_api_defs.h | 1 + repair/agbtree.c | 22 +++++++++++----- repair/bulkload.c | 63 +++++++++++++++++++++++++++++++++------------- repair/bulkload.h | 12 +++++---- repair/phase5.c | 2 + 5 files changed, 69 insertions(+), 31 deletions(-)
libxfs/libxfs_api_defs.h +1 -0
@@ -149,6 +149,7 @@#define xfs_log_sb libxfs_log_sb#define xfs_mode_to_ftype libxfs_mode_to_ftype#define xfs_perag_get libxfs_perag_get +#define xfs_perag_hold libxfs_perag_hold#define xfs_perag_put libxfs_perag_put#define xfs_prealloc_blocks libxfs_prealloc_blocks
repair/agbtree.c +15 -7
@@ -77,13 +77,17 @@ reserve_agblocks( uint32_t nr_blocks) {struct extent_tree_node *ext_ptr; +struct xfs_perag *pag;uint32_t blocks_allocated =0 ;uint32_t len;int error; + pag =libxfs_perag_get (mp, agno); +if (!pag) +do_error (_ ("could not open perag structure for agno 0x%x\n" ), + agno); +while (blocks_allocated < nr_blocks) { -xfs_fsblock_t fsbno; -/* * Grab the smallest extent and use it up, then get the * next smallest. This mimics the init_*_cursor code.
@@ -94,8 +98,8 @@ reserve_agblocks(/* Use up the extent we've got. */ len =min (ext_ptr->ex_blockcount, nr_blocks - blocks_allocated); - fsbno =XFS_AGB_TO_FSB (mp, agno, ext_ptr->ex_startblock); - error =bulkload_add_blocks (&btr->newbt, fsbno, len); + error =bulkload_add_extent (&btr->newbt, pag, + ext_ptr->ex_startblock, len);if (error)do_error (_ ("could not set up btree reservation: %s\n" ),strerror (-error));
@@ -113,6 +117,7 @@ reserve_agblocks(fprintf (stderr,"blocks_allocated = %d\n" , blocks_allocated);#endif +libxfs_perag_put (pag);return blocks_allocated == nr_blocks; }
@@ -155,18 +160,21 @@ finish_rebuild(int error;for_each_bulkload_reservation (&btr->newbt, resv, n) { +xfs_fsblock_t fsbno; +if (resv->used == resv->len)continue ; - error =bitmap_set (lost_blocks, resv->fsbno + resv->used, - resv->len - resv->used); + fsbno =XFS_AGB_TO_FSB (mp, resv->pag->pag_agno, + resv->agbno + resv->used); + error =bitmap_set (lost_blocks, fsbno, resv->len - resv->used);if (error)do_error (_ ("Insufficient memory saving lost blocks, err=%d.\n" ), error); resv->used = resv->len; } -bulkload_destroy (&btr->newbt,0 ); +bulkload_commit (&btr->newbt); }/*
repair/bulkload.c +45 -18
107 lines not on this page · raw message
repair/bulkload.h +7 -5
@@ -17,8 +17,10 @@ struct bulkload_resv {/* Link to list of extents that we've reserved. */ struct list_head list; -/* FSB of the block we reserved. */ -xfs_fsblock_t fsbno; +struct xfs_perag *pag; + +/* AG block of the block we reserved. */ +xfs_agblock_t agbno;/* Length of the reservation. */ xfs_extlen_t len;
@@ -51,11 +53,11 @@ struct bulkload {void bulkload_init_ag (struct bulkload *bkl,struct repair_ctx *sc,const struct xfs_owner_info *oinfo); -int bulkload_add_blocks (struct bulkload *bkl,xfs_fsblock_t fsbno, -xfs_extlen_t len); -void bulkload_destroy (struct bulkload *bkl,int error);int bulkload_claim_block (struct xfs_btree_cur *cur,struct bulkload *bkl,union xfs_btree_ptr *ptr); +int bulkload_add_extent (struct bulkload *bkl,struct xfs_perag *pag, +xfs_agblock_t agbno,xfs_extlen_t len); +void bulkload_commit (struct bulkload *bkl);void bulkload_estimate_ag_slack (struct repair_ctx *sc,struct xfs_btree_bload *bload,unsigned int free);
repair/phase5.c +1 -1
@@ -194,7 +194,7 @@ fill_agfl(for_each_bulkload_reservation (&btr->newbt, resv, n) {xfs_agblock_t bno; - bno =XFS_FSB_TO_AGBNO (mp, resv->fsbno + resv->used); + bno = resv->agbno + resv->used;while (resv->used < resv->len && *agfl_idx <libxfs_agfl_size (mp)) { agfl_bnos[(*agfl_idx)++] =cpu_to_be32 (bno++);
PATCH 3/3 xfs_repair: rebuild block mappings from rmapbt data
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use rmap records to rebuild corrupt inode forks instead of zapping the whole inode if we think the rmap data is reasonably sane.
--- include/xfs_trans.h | 2 libfrog/util.h | 5 libxfs/libxfs_api_defs.h | 15 + libxfs/trans.c | 48 +++ repair/Makefile | 2 repair/agbtree.c | 2 repair/bmap_repair.c | 748 ++++++++++++++++++++++++++++++++++++++++++++++ repair/bmap_repair.h | 13 + repair/bulkload.c | 205 ++++++++++++- repair/bulkload.h | 24 + repair/dinode.c | 54 +++ repair/rmap.c | 2 repair/rmap.h | 1 13 files changed, 1110 insertions(+), 11 deletions(-) create mode 100644 repair/bmap_repair.c create mode 100644 repair/bmap_repair.h
include/xfs_trans.h +2 -0
@@ -98,6 +98,8 @@ int libxfs_trans_alloc_rollable(struct xfs_mount *mp, uint blocks,int libxfs_trans_alloc_empty (struct xfs_mount *mp,struct xfs_trans **tpp);int libxfs_trans_commit (struct xfs_trans *);void libxfs_trans_cancel (struct xfs_trans *); +int libxfs_trans_reserve_more (struct xfs_trans *tp,uint blocks, +uint rtextents);/* cancel dfops associated with a transaction */ void xfs_defer_cancel (struct xfs_trans *);
libfrog/util.h +5 -0
@@ -8,4 +8,9 @@unsigned int log2_roundup (unsigned int i); +#define min_t (type,x,y) \ + ({ type __x = (x); type __y = (y); __x < __y ? __x: __y; }) +#define max_t (type,x,y) \ + ({ type __x = (x); type __y = (y); __x > __y ? __x: __y; }) +#endif /* __LIBFROG_UTIL_H__ */
libxfs/libxfs_api_defs.h +14 -1
@@ -32,7 +32,7 @@#define xfs_alloc_fix_freelist libxfs_alloc_fix_freelist#define xfs_alloc_min_freelist libxfs_alloc_min_freelist#define xfs_alloc_read_agf libxfs_alloc_read_agf -#define xfs_alloc_vextent libxfs_alloc_vextent +#define xfs_alloc_vextent_start_ag libxfs_alloc_vextent_start_ag#define xfs_ascii_ci_hashname libxfs_ascii_ci_hashname
@@ -44,11 +44,18 @@#define xfs_attr_shortform_verify libxfs_attr_shortform_verify#define __xfs_bmap_add_free __libxfs_bmap_add_free +#define xfs_bmap_validate_extent libxfs_bmap_validate_extent#define xfs_bmapi_read libxfs_bmapi_read +#define xfs_bmapi_remap libxfs_bmapi_remap#define xfs_bmapi_write libxfs_bmapi_write#define xfs_bmap_last_offset libxfs_bmap_last_offset +#define xfs_bmbt_calc_size libxfs_bmbt_calc_size +#define xfs_bmbt_commit_staged_btree libxfs_bmbt_commit_staged_btree +#define xfs_bmbt_disk_get_startoff libxfs_bmbt_disk_get_startoff +#define xfs_bmbt_disk_set_all libxfs_bmbt_disk_set_all#define xfs_bmbt_maxlevels_ondisk libxfs_bmbt_maxlevels_ondisk#define xfs_bmbt_maxrecs libxfs_bmbt_maxrecs +#define xfs_bmbt_stage_cursor libxfs_bmbt_stage_cursor#define xfs_bmdr_maxrecs libxfs_bmdr_maxrecs#define xfs_btree_bload libxfs_btree_bload
@@ -117,6 +124,7 @@#define xfs_finobt_calc_reserves libxfs_finobt_calc_reserves#define xfs_free_extent libxfs_free_extent +#define xfs_free_extent_later libxfs_free_extent_later#define xfs_free_perag libxfs_free_perag#define xfs_fs_geometry libxfs_fs_geometry#define xfs_highbit32 libxfs_highbit32
@@ -127,7 +135,10 @@#define xfs_ialloc_read_agi libxfs_ialloc_read_agi#define xfs_idata_realloc libxfs_idata_realloc#define xfs_idestroy_fork libxfs_idestroy_fork +#define xfs_iext_first libxfs_iext_first +#define xfs_iext_insert_raw libxfs_iext_insert_raw#define xfs_iext_lookup_extent libxfs_iext_lookup_extent +#define xfs_iext_next libxfs_iext_next#define xfs_ifork_zap_attr libxfs_ifork_zap_attr#define xfs_imap_to_bp libxfs_imap_to_bp#define xfs_initialize_perag libxfs_initialize_perag
@@ -174,10 +185,12 @@#define xfs_rmapbt_stage_cursor libxfs_rmapbt_stage_cursor#define xfs_rmap_compare libxfs_rmap_compare#define xfs_rmap_get_rec libxfs_rmap_get_rec +#define xfs_rmap_ino_bmbt_owner libxfs_rmap_ino_bmbt_owner#define xfs_rmap_irec_offset_pack libxfs_rmap_irec_offset_pack#define xfs_rmap_irec_offset_unpack libxfs_rmap_irec_offset_unpack#define xfs_rmap_lookup_le libxfs_rmap_lookup_le#define xfs_rmap_lookup_le_range libxfs_rmap_lookup_le_range +#define xfs_rmap_query_all libxfs_rmap_query_all#define xfs_rmap_query_range libxfs_rmap_query_range#define xfs_rtbitmap_getword libxfs_rtbitmap_getword
libxfs/trans.c +48 -0
51 lines not on this page · raw message
repair/Makefile +2 -0
@@ -16,6 +16,7 @@ HFILES = \ avl.h \ bulkload.h \ bmap.h \ + bmap_repair.h \ btree.h \ da_util.h \ dinode.h \
@@ -41,6 +42,7 @@ CFILES = \ avl.c \ bulkload.c \ bmap.c \ + bmap_repair.c \ btree.c \ da_util.c \ dino_chunks.c \
repair/agbtree.c +1 -1
@@ -22,7 +22,7 @@ init_rebuild( {memset (btr,0 ,sizeof (struct bt_rebuild )); -bulkload_init_ag (&btr->newbt, sc, oinfo); +bulkload_init_ag (&btr->newbt, sc, oinfo,NULLFSBLOCK ); btr->bload.max_dirty =XFS_B_TO_FSBT (sc->mp,256U <<10 );/* 256K */ bulkload_estimate_ag_slack (sc, &btr->bload, est_agfreeblocks); }
repair/bmap_repair.c new file +748 -0
748 lines not on this page · raw message
repair/bmap_repair.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (c) 2019-2024 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +#ifndef REBUILD_H_ +#define REBUILD_H_ + +int rebuild_bmap (struct xfs_mount *mp,xfs_ino_t ino,int whichfork, +unsigned long nr_extents,struct xfs_buf **ino_bpp, +struct xfs_dinode **dinop,int *dirty); + +#endif /* REBUILD_H_ */
repair/bulkload.c +201 -4
232 lines not on this page · raw message
repair/bulkload.h +21 -3
@@ -8,9 +8,17 @@extern int bload_leaf_slack;extern int bload_node_slack; +/* +* This is the maximum number of deferred extent freeing item extents (EFIs) +* that we'll attach to a transaction without rolling the transaction to avoid +* overrunning a tr_itruncate reservation. +*/ +#define XREP_MAX_ITRUNCATE_EFIS (128)struct repair_ctx {struct xfs_mount *mp; +struct xfs_inode *ip; +struct xfs_trans *tp; };struct bulkload_resv {
@@ -36,7 +44,10 @@ struct bulkload {struct list_head resv_list;/* Fake root for new btree. */ -struct xbtree_afakeroot afake; +union { +struct xbtree_afakeroot afake; +struct xbtree_ifakeroot ifake; + };/* rmap owner of these blocks */ struct xfs_owner_info oinfo;
@@ -44,6 +55,9 @@ struct bulkload {/* The last reservation we allocated from. */ struct bulkload_resv *last_resv; +/* Hint as to where we should allocate blocks. */ +xfs_fsblock_t alloc_hint; +/* Number of blocks reserved via resv_list. */ unsigned int nr_reserved; };
@@ -52,12 +66,16 @@ struct bulkload {list_for_each_entry_safe ((resv), (n), &(bkl)->resv_list, list) void bulkload_init_ag(struct bulkload *bkl, struct repair_ctx *sc, - const struct xfs_owner_info *oinfo); + const struct xfs_owner_info *oinfo, xfs_fsblock_t alloc_hint); +void bulkload_init_inode (struct bulkload *bkl,struct repair_ctx *sc, +int whichfork,const struct xfs_owner_info *oinfo);int bulkload_claim_block (struct xfs_btree_cur *cur,struct bulkload *bkl,union xfs_btree_ptr *ptr);int bulkload_add_extent (struct bulkload *bkl,struct xfs_perag *pag,xfs_agblock_t agbno,xfs_extlen_t len); -void bulkload_commit (struct bulkload *bkl); +int bulkload_alloc_file_blocks (struct bulkload *bkl,uint64_t nr_blocks); +void bulkload_cancel (struct bulkload *bkl); +int bulkload_commit (struct bulkload *bkl);void bulkload_estimate_ag_slack (struct repair_ctx *sc,struct xfs_btree_bload *bload,unsigned int free);
repair/dinode.c +53 -1
114 lines not on this page · raw message
repair/rmap.c +1 -1
@@ -33,7 +33,7 @@ struct xfs_ag_rmap { };static struct xfs_ag_rmap *ag_rmaps; -static bool rmapbt_suspect; +bool rmapbt_suspect;static bool refcbt_suspect;static inline int rmap_compare (const void *a,const void *b)
repair/rmap.h +1 -0
@@ -7,6 +7,7 @@#define RMAP_H_ extern bool collect_rmaps; +extern bool rmapbt_suspect;extern bool rmap_needs_work (struct xfs_mount *);
PATCH 9/18 xfs_repair: support more than 4 billion records
In reply to Darrick J. WongHi all,
I started looking through all the places where XFS has to deal with the rc_refcount attribute of refcount records, and noticed that offline repair doesn't handle the situation where there are more than 2^32 reverse mappings in an AG, or that there are more than 2^32 owners of a particular piece of AG space. I've estimated that it would take several months to produce a filesystem with this many records, but we really ought to do better at handling them than crashing or (worse) not crashing and writing out corrupt btrees due to integer truncation.
Once I started using the bmap_inflate debugger command to create extreme reflink scenarios, I noticed that the memory usage of xfs_repair was astronomical. This I observed to be due to the fact that it allocates a single huge block mapping array for all files on the system, even though it only uses that array for data and attr forks that map metadata blocks (e.g. directories, xattrs, symlinks) and does not use it for regular data files.
So I got rid of the 2^31-1 limits on the block map array and turned off the block mapping for regular data files. This doesn't answer the question of what to do if there are a lot of extents, but it kicks the can down the road until someone creates a maximally sized xattr tree, which so far nobody's ever stuck to long enough to complain about.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-support-4bn-records
---
Commits in this patchset:
* xfs_db: add a bmbt inflation command
* xfs_repair: slab and bag structs need to track more than 2^32 items
* xfs_repair: support more than 2^32 rmapbt records per AG
* xfs_repair: support more than 2^32 owners per physical block
* xfs_repair: clean up lock resources
* xfs_repair: constrain attr fork extent count
* xfs_repair: don't create block maps for data files
* xfs_repair: support more than INT_MAX block maps
--- db/Makefile | 65 +++++- db/bmap_inflate.c | 551 +++++++++++++++++++++++++++++++++++++++++++++++++++++ db/command.c | 1 db/command.h | 1 man/man8/xfs_db.8 | 23 ++ repair/bmap.c | 23 +- repair/bmap.h | 7 - repair/dinode.c | 18 +- repair/dir2.c | 2 repair/incore.c | 9 + repair/rmap.c | 25 +- repair/rmap.h | 4 repair/slab.c | 36 ++- repair/slab.h | 36 ++- 14 files changed, 725 insertions(+), 76 deletions(-) create mode 100644 db/bmap_inflate.c
PATCH 1/8 xfs_db: add a bmbt inflation command
From: Darrick J. Wong <[email protected]>
Add a command to xfs_db to clone a data fork mapping over and over again. This will make it easier to exercise really high sharing counts.
--- db/Makefile | 65 +++++- db/bmap_inflate.c | 551 +++++++++++++++++++++++++++++++++++++++++++++++++++++ db/command.c | 1 db/command.h | 1 man/man8/xfs_db.8 | 23 ++ 5 files changed, 633 insertions(+), 8 deletions(-) create mode 100644 db/bmap_inflate.c
db/Makefile +57 -8
@@ -7,14 +7,63 @@ include $(TOPDIR)/include/builddefs LTCOMMAND = xfs_db -HFILES = addr.h agf.h agfl.h agi.h attr.h attrshort.h bit.h block.h bmap.h \ - btblock.h bmroot.h check.h command.h crc.h debug.h \ - dir2.h dir2sf.h dquot.h echo.h faddr.h field.h \ - flist.h fprint.h frag.h freesp.h hash.h help.h init.h inode.h input.h \ - io.h logformat.h malloc.h metadump.h output.h print.h quit.h sb.h \ - sig.h strvec.h text.h type.h write.h attrset.h symlink.h fsmap.h \ - fuzz.h obfuscate.h -CFILES = $(HFILES:.h=.c) btdump.c btheight.c convert.c info.c iunlink.c namei.c \ +HFILES = \ + addr.h \ + agf.h \ + agfl.h \ + agi.h \ + attr.h \ + attrset.h \ + attrshort.h \ + bit.h \ + block.h \ + bmap.h \ + bmroot.h \ + btblock.h \ + check.h \ + command.h \ + crc.h \ + debug.h \ + dir2.h \ + dir2sf.h \ + dquot.h \ + echo.h \ + faddr.h \ + field.h \ + flist.h \ + fprint.h \ + frag.h \ + freesp.h \ + fsmap.h \ + fuzz.h \ + hash.h \ + help.h \ + init.h \ + inode.h \ + input.h \ + io.h \ + logformat.h \ + malloc.h \ + metadump.h \ + obfuscate.h \ + output.h \ + print.h \ + quit.h \ + sb.h \ + sig.h \ + strvec.h \ + symlink.h \ + text.h \ + type.h \ + write.h +CFILES = $(HFILES:.h=.c) \ + bmap_inflate.c \ + btdump.c \ + btheight.c \ + convert.c \ + info.c \ + iunlink.c \ + namei.c \ timelimit.c LSRCFILES = xfs_admin.sh xfs_ncheck.sh xfs_metadump.sh
db/bmap_inflate.c new file +551 -0
551 lines not on this page · raw message
db/command.c +1 -0
@@ -142,4 +142,5 @@ init_commands(void)fuzz_init ();timelimit_init ();iunlink_init (); +bmapinflate_init (); }
db/command.h +1 -0
@@ -35,3 +35,4 @@ extern void btheight_init(void);extern void timelimit_init (void );extern void namei_init (void );extern void iunlink_init (void ); +extern void bmapinflate_init (void );
man/man8/xfs_db.8 +23 -0
@@ -388,6 +388,29 @@ and options are used to select the attribute or data area of the inode, if neither option is given then both areas are shown. .TP +.BI "bmapinflate [\-d " dirty_bytes "] [-e] [\-n " nr "] +Duplicates the first data fork mapping this many times, as if the mapping had +been repeatedly reflinked. +This is an expert-mode command for exercising high-refcount filesystems only. +Existing data fork mappings will be forgotten and the refcount btree will not +be updated. +This command leaves at least the refcount btree and the inode inconsistent; +.B xfs_repair +must be run afterwards. +.RS 1.0i +.TP 0.4i +.B \-d +Constrain the memory consumption of new dirty btree blocks to this quantity. +Defaults to 60MiB. +.TP 0.4i +.B \-e +Estimate the number of blocks and height of the new data fork mapping +structure and exit without changing anything. +.TP 0.4i +.B \-n +Create this many copies of the first mapping. +.RE +.TP .B btdump [-a] [-i] If the cursor points to a btree node, dump the btree from that block downward. If instead the cursor points to an inode, dump the data fork block mapping btree if there is one.