xfs: Parent Pointers
Hi everyone,
This deluge contains all of the additions to the parent pointers patchset that I've been working since last month's deluge. The kernel and xfsprogs patchsets are based on Allison's v10 tag from last week; the fstests patches are merely a part of my development tree. To recap Allison's cover letter:
"The goal of this patch set is to add a parent pointer attribute to each inode. The attribute name containing the parent inode, generation, and directory offset, while the attribute value contains the file name. This feature will enable future optimizations for online scrub, shrink, nfs handles, verity, or any other feature that could make use of quickly deriving an inodes path from the mount point."
v10r1d2 rebases everything against 6.3-rc2. I still want to remove the diroffset from the ondisk parent pointer, but for v10 I've replaced the sha512 hashing code with modifications to the xattr code to support lookups based on name *and* value. With that working, we can encode parent pointers like this:
(parent_ino, parent_gen, name[])
xattr lookups still work correctly, and repair doesn't have to deal with keeping the diroffsets in sync if the directory gets rebuilt. With this change applied, I'm ready to weave my new changes into Allison's v10 and call parent pointers done. :)
The online directory and parent pointer code are exactly the same as the v9r2d1 release, so I'm eliding that and everything that was in Allison's recent v10 patchset. IOWs, this deluge includes only the bug fixes I've made to parent pointers, the updates I've made to the ondisk format, and the necessary changes to fstests to get everything to pass.
If you want to pull the whole thing, use these links:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-drop-unnecessary
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-drop-unnecessary
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key
Allison: Could you please resync libxfs in the following patches under
https://github.com/allisonhenderson/xfsprogs/commits/xfsprogs_new_pptrs_v10
please?
xfsprogs: add parent pointer support to attribute code
xfsprogs: extend transaction reservations for parent attributes
xfsprogs: parent pointer attribute creation
xfsprogs: remove parent pointers in unlink
xfsprogs: Add parent pointers to rename
xfsprogs: move/add parent pointer validators to xfs_parent
There are discrepancies between the two, which makes ./tools/libxfs-diff unhappy. Or, if you want me to merge my ondisk format changes into my branches, I'll put out v11 with everything taken care of.
--D
cover xfs: bug fixes for parent pointers
Hi all,
This series contains the accumulated bug fixes from Darrick to make fstests pass and online repair work.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-bugfixes
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-bugfixes
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-bugfixes
--- fs/xfs/libxfs/xfs_da_format.h | 11 ++++++ fs/xfs/libxfs/xfs_fs.h | 69 +++++++++++++++++-------------------- fs/xfs/libxfs/xfs_parent.c | 47 ++++++++++++++++++++++--- fs/xfs/libxfs/xfs_parent.h | 24 ++++++------- fs/xfs/xfs_inode.c | 16 ++++----- fs/xfs/xfs_ioctl.c | 47 ++++++++++++------------- fs/xfs/xfs_ondisk.h | 4 +- fs/xfs/xfs_parent_utils.c | 61 +++++++++++++++++---------------- fs/xfs/xfs_parent_utils.h | 8 ++-- fs/xfs/xfs_symlink.c | 2 + fs/xfs/xfs_trace.c | 1 + fs/xfs/xfs_trace.h | 76 ++++++++++++++++++++++++++++++++++++++++- 12 files changed, 238 insertions(+), 128 deletions(-)
PATCH 1/7 xfs: validate parent pointer xattrs in getparent
From: Darrick J. Wong <[email protected]>
Actually validate the parent pointer xattr before we try to export it to userspace.
--- fs/xfs/xfs_parent_utils.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
fs/xfs/xfs_parent_utils.c +6 -8
@@ -49,6 +49,7 @@ xfs_getparent_listent(struct xfs_getparent_ctx *gp;struct xfs_pptr_info *ppi;struct xfs_parent_ptr *pptr; +struct xfs_parent_name_rec *rec = (void *)name;struct xfs_parent_name_irec *irec;struct xfs_mount *mp = context->dp->i_mount;int arraytop;
@@ -62,19 +63,16 @@ xfs_getparent_listent(return ;/* -* Report corruption for xattrs with any other flag set, or for a -* parent pointer that has a remote value. The attr list functions -* filtered any INCOMPLETE attrs for us. +* Report corruption for anything that doesn't look like a parent +* pointer. The attr list functions filtered out INCOMPLETE attrs. */ -if (XFS_IS_CORRUPT (mp, -hweight32 (flags &XFS_ATTR_NSP_ONDISK_MASK ) >1 ) || -XFS_IS_CORRUPT (mp, value ==NULL )) { +if (XFS_IS_CORRUPT (mp, !xfs_parent_namecheck (mp, rec, namelen, flags)) || +XFS_IS_CORRUPT (mp, !xfs_parent_valuecheck (mp, value, valuelen))) { context->seen_enough = -EFSCORRUPTED ;return ; } -xfs_parent_irec_from_disk (&gp->pptr_irec, (void *)name, value, - valuelen); +xfs_parent_irec_from_disk (&gp->pptr_irec, rec, value, valuelen);/* * We found a parent pointer, but we've filled up the buffer. Signal
PATCH 2/7 xfs: rename xfs_pptr_info to xfs_getparents
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Rename the head structure of the parent pointer ioctl to match the name of the ioctl (XFS_IOC_GETPARENTS).
--- fs/xfs/libxfs/xfs_fs.h | 45 +++++++++++++++++++++++--------------------- fs/xfs/xfs_ioctl.c | 46 +++++++++++++++++++++++---------------------- fs/xfs/xfs_ondisk.h | 2 +- fs/xfs/xfs_parent_utils.c | 26 +++++++++++++------------ fs/xfs/xfs_parent_utils.h | 8 ++++---- 5 files changed, 64 insertions(+), 63 deletions(-)
fs/xfs/libxfs/xfs_fs.h +23 -22
@@ -752,19 +752,20 @@ struct xfs_scrub_metadata { XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED)#define XFS_SCRUB_FLAGS_ALL (XFS_SCRUB_FLAGS_IN | XFS_SCRUB_FLAGS_OUT) -#define XFS_PPTR_MAXNAMELEN 256 +#define XFS_GETPARENTS_MAXNAMELEN 256/* return parents of the handle, not the open fd */ -#define XFS_PPTR_IFLAG_HANDLE (1U << 0) +#define XFS_GETPARENTS_IFLAG_HANDLE (1U << 0)/* target was the root directory */ -#define XFS_PPTR_OFLAG_ROOT (1U << 1) +#define XFS_GETPARENTS_OFLAG_ROOT (1U << 1)/* Cursor is done iterating pptrs */ -#define XFS_PPTR_OFLAG_DONE (1U << 2) +#define XFS_GETPARENTS_OFLAG_DONE (1U << 2) -#define XFS_PPTR_FLAG_ALL (XFS_PPTR_IFLAG_HANDLE | XFS_PPTR_OFLAG_ROOT | \ - XFS_PPTR_OFLAG_DONE) +#define XFS_GETPARENTS_FLAG_ALL (XFS_GETPARENTS_IFLAG_HANDLE | \ + XFS_GETPARENTS_OFLAG_ROOT | \ + XFS_GETPARENTS_OFLAG_DONE)/* Get an inode parent pointer through ioctl */ struct xfs_parent_ptr {
@@ -776,49 +777,49 @@ struct xfs_parent_ptr { };/* Iterate through an inodes parent pointers */ -struct xfs_pptr_info { -/* File handle, if XFS_PPTR_IFLAG_HANDLE is set */ -struct xfs_handle pi_handle; +struct xfs_getparents { +/* File handle, if XFS_GETPARENTS_IFLAG_HANDLE is set */ +struct xfs_handle gp_handle;/* * Structure to track progress in iterating the parent pointers. * Must be initialized to zeroes before the first ioctl call, and * not touched by callers after that. */ -struct xfs_attrlist_cursor pi_cursor; +struct xfs_attrlist_cursor gp_cursor; -/* Operational flags: XFS_PPTR_*FLAG* */ -__u32 pi_flags; +/* Operational flags: XFS_GETPARENTS_*FLAG* */ +__u32 gp_flags;/* Must be set to zero */ -__u32 pi_reserved; +__u32 gp_reserved;/* size of the trailing buffer in bytes */ -__u32 pi_ptrs_size; +__u32 gp_ptrs_size;/* # of entries filled in (output) */ -__u32 pi_count; +__u32 gp_count;/* Must be set to zero */ -__u64 pi_reserved2[5 ]; +__u64 gp_reserved2[5 ];/* Byte offset of each record within the buffer */ -__u32 pi_offsets[]; +__u32 gp_offsets[]; };static inline size_t -xfs_pptr_info_sizeof (int nr_ptrs) +xfs_getparents_sizeof (int nr_ptrs) { -return sizeof (struct xfs_pptr_info ) + +return sizeof (struct xfs_getparents ) + (nr_ptrs *sizeof (struct xfs_parent_ptr )); }static inline struct xfs_parent_ptr * -xfs_ppinfo_to_pp ( -struct xfs_pptr_info *info, +xfs_getparents_rec ( +struct xfs_getparents *info,int idx) { -return (struct xfs_parent_ptr *)((char *)info + info->pi_offsets[idx]); +return (struct xfs_parent_ptr *)((char *)info + info->gp_offsets[idx]); }/*
fs/xfs/xfs_ioctl.c +23 -23
114 lines not on this page · raw message
fs/xfs/xfs_ondisk.h +1 -1
@@ -152,7 +152,7 @@ xfs_check_ondisk_structs(void)/* parent pointer ioctls */ XFS_CHECK_STRUCT_SIZE(struct xfs_parent_ptr, 24); - XFS_CHECK_STRUCT_SIZE(struct xfs_pptr_info, 96); + XFS_CHECK_STRUCT_SIZE(struct xfs_getparents, 96);/* * The v5 superblock format extended several v4 header structures with
fs/xfs/xfs_parent_utils.c +13 -13
@@ -26,7 +26,7 @@struct xfs_getparent_ctx {struct xfs_attr_list_context context;struct xfs_parent_name_irec pptr_irec; -struct xfs_pptr_info *ppi; +struct xfs_getparents *ppi; }; static inline unsigned int
@@ -47,7 +47,7 @@ xfs_getparent_listent( int valuelen) {struct xfs_getparent_ctx *gp; -struct xfs_pptr_info *ppi; +struct xfs_getparents *ppi;struct xfs_parent_ptr *pptr;struct xfs_parent_name_rec *rec = (void *)name;struct xfs_parent_name_irec *irec;
@@ -79,7 +79,7 @@ xfs_getparent_listent(* to the caller that we did /not/ reach the end of the parent pointer * recordset. */ - arraytop =xfs_getparents_arraytop (ppi, ppi->pi_count +1 ); + arraytop =xfs_getparents_arraytop (ppi, ppi->gp_count +1 ); context->firstu -=xfs_getparents_rec_sizeof (irec);if (context->firstu < arraytop) { context->seen_enough =1 ;
@@ -87,8 +87,8 @@ xfs_getparent_listent( }/* Format the parent pointer directly into the caller buffer. */ - ppi->pi_offsets[ppi->pi_count] = context->firstu; - pptr =xfs_ppinfo_to_pp (ppi, ppi->pi_count); + ppi->gp_offsets[ppi->gp_count] = context->firstu; + pptr =xfs_getparents_rec (ppi, ppi->gp_count); pptr->xpp_ino = irec->p_ino; pptr->xpp_gen = irec->p_gen; pptr->xpp_diroffset = irec->p_diroffset;
@@ -96,14 +96,14 @@ xfs_getparent_listent(memcpy (pptr->xpp_name, irec->p_name, irec->p_namelen); pptr->xpp_name[irec->p_namelen] =0 ; - ppi->pi_count++; + ppi->gp_count++; }/* Retrieve the parent pointers for a given inode. */ int xfs_getparent_pointers (struct xfs_inode *ip, -struct xfs_pptr_info *ppi) +struct xfs_getparents *ppi) {struct xfs_getparent_ctx *gp;int error;
@@ -115,13 +115,13 @@ xfs_getparent_pointers( gp->context.dp = ip; gp->context.resynch =1 ; gp->context.put_listent = xfs_getparent_listent; - gp->context.bufsize =round_down (ppi->pi_ptrs_size,sizeof (uint32_t )); + gp->context.bufsize =round_down (ppi->gp_ptrs_size,sizeof (uint32_t )); gp->context.firstu = gp->context.bufsize;/* Copy the cursor provided by caller */ -memcpy (&gp->context.cursor, &ppi->pi_cursor, +memcpy (&gp->context.cursor, &ppi->gp_cursor,sizeof (struct xfs_attrlist_cursor )); - ppi->pi_count =0 ; + ppi->gp_count =0 ; error =xfs_attr_list (&gp->context);if (error)
@@ -133,17 +133,17 @@ xfs_getparent_pointers(/* Is this the root directory? */ if (ip->i_ino == ip->i_mount->m_sb.sb_rootino) - ppi->pi_flags |=XFS_PPTR_OFLAG_ROOT ; + ppi->gp_flags |=XFS_GETPARENTS_OFLAG_ROOT ;/* * If we did not run out of buffer space, then we reached the end of * the pptr recordset, so set the DONE flag. */ if (gp->context.seen_enough ==0 ) - ppi->pi_flags |=XFS_PPTR_OFLAG_DONE ; + ppi->gp_flags |=XFS_GETPARENTS_OFLAG_DONE ;/* Update the caller with the current cursor position */ -memcpy (&ppi->pi_cursor, &gp->context.cursor, +memcpy (&ppi->gp_cursor, &gp->context.cursor,sizeof (struct xfs_attrlist_cursor )); out_free:kfree (gp);
fs/xfs/xfs_parent_utils.h +4 -4
@@ -8,13 +8,13 @@static inline unsigned int xfs_getparents_arraytop ( -const struct xfs_pptr_info *ppi, +const struct xfs_getparents *ppi,unsigned int nr) { -return sizeof (struct xfs_pptr_info ) + - (nr *sizeof (ppi->pi_offsets[0 ])); +return sizeof (struct xfs_getparents ) + + (nr *sizeof (ppi->gp_offsets[0 ])); } -int xfs_getparent_pointers (struct xfs_inode *ip,struct xfs_pptr_info *ppi); +int xfs_getparent_pointers (struct xfs_inode *ip,struct xfs_getparents *ppi);#endif /* __XFS_PARENT_UTILS_H__ */
PATCH 3/7 xfs: rename xfs_parent_ptr
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Change the name to xfs_getparents_rec so that the name matches the head structure.
--- fs/xfs/libxfs/xfs_fs.h | 20 ++++++++++---------- fs/xfs/xfs_ioctl.c | 6 +++--- fs/xfs/xfs_ondisk.h | 2 +- fs/xfs/xfs_parent_utils.c | 16 ++++++++-------- 4 files changed, 22 insertions(+), 22 deletions(-)
fs/xfs/libxfs/xfs_fs.h +10 -10
@@ -768,12 +768,12 @@ struct xfs_scrub_metadata { XFS_GETPARENTS_OFLAG_DONE)/* Get an inode parent pointer through ioctl */ -struct xfs_parent_ptr { -__u64 xpp_ino;/* Inode */ -__u32 xpp_gen;/* Inode generation */ -__u32 xpp_diroffset;/* Directory offset */ -__u64 xpp_rsvd;/* Reserved */ -__u8 xpp_name[];/* File name */ +struct xfs_getparents_rec { +__u64 gpr_ino;/* Inode */ +__u32 gpr_gen;/* Inode generation */ +__u32 gpr_diroffset;/* Directory offset */ +__u64 gpr_rsvd;/* Reserved */ +__u8 gpr_name[];/* File name */ };/* Iterate through an inodes parent pointers */
@@ -811,15 +811,15 @@ static inline size_t xfs_getparents_sizeof(int nr_ptrs) {return sizeof (struct xfs_getparents ) + - (nr_ptrs *sizeof (struct xfs_parent_ptr )); + (nr_ptrs *sizeof (struct xfs_getparents_rec )); } -static inline struct xfs_parent_ptr * +static inline struct xfs_getparents_rec *xfs_getparents_rec (struct xfs_getparents *info,int idx) { -return (struct xfs_parent_ptr *)((char *)info + info->gp_offsets[idx]); +return (struct xfs_getparents_rec *)((char *)info + info->gp_offsets[idx]); }/*
@@ -867,7 +867,7 @@ xfs_getparents_rec(/* XFS_IOC_GETFSMAP ------ hoisted 59 */ #define XFS_IOC_SCRUB_METADATA _IOWR('X', 60, struct xfs_scrub_metadata)#define XFS_IOC_AG_GEOMETRY _IOWR('X', 61, struct xfs_ag_geometry) -#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_parent_ptr) +#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_getparents_rec)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_ioctl.c +3 -3
@@ -1681,10 +1681,10 @@ xfs_ioc_scrub_metadata(* IOCTL routine to get the parent pointers of an inode and return it to user * space. Caller must pass a buffer space containing a struct xfs_getparents, * followed by a region large enough to contain an array of struct -* xfs_parent_ptr of a size specified in gp_ptrs_size. If the inode contains +* xfs_getparents_rec of a size specified in gp_ptrs_size. If the inode contains * more parent pointers than can fit in the buffer space, caller may re-call * the function using the returned gp_cursor to resume iteration. The -* number of xfs_parent_ptr returned will be stored in gp_ptrs_count. +* number of xfs_getparents_rec returned will be stored in gp_ptrs_count. * * Returns 0 on success or non-zero on failure */
@@ -1699,7 +1699,7 @@ xfs_ioc_get_parent_pointer(struct xfs_inode *call_ip = file_ip;struct xfs_mount *mp = file_ip->i_mount;void __user *o_pptr; -struct xfs_parent_ptr *i_pptr; +struct xfs_getparents_rec *i_pptr;unsigned int bytes;if (!capable (CAP_SYS_ADMIN ))
fs/xfs/xfs_ondisk.h +1 -1
@@ -151,7 +151,7 @@ xfs_check_ondisk_structs(void) XFS_CHECK_OFFSET(struct xfs_efi_log_format_64, efi_extents, 16);/* parent pointer ioctls */ - XFS_CHECK_STRUCT_SIZE(struct xfs_parent_ptr, 24); + XFS_CHECK_STRUCT_SIZE(struct xfs_getparents_rec, 24); XFS_CHECK_STRUCT_SIZE(struct xfs_getparents, 96);/*
fs/xfs/xfs_parent_utils.c +8 -8
@@ -33,7 +33,7 @@ static inline unsigned intxfs_getparents_rec_sizeof ( const struct xfs_parent_name_irec *irec) { -return round_up (sizeof (struct xfs_parent_ptr ) + irec->p_namelen +1 , +return round_up (sizeof (struct xfs_getparents_rec ) + irec->p_namelen +1 ,sizeof (uint32_t )); }
@@ -48,7 +48,7 @@ xfs_getparent_listent( {struct xfs_getparent_ctx *gp;struct xfs_getparents *ppi; -struct xfs_parent_ptr *pptr; +struct xfs_getparents_rec *pptr;struct xfs_parent_name_rec *rec = (void *)name;struct xfs_parent_name_irec *irec;struct xfs_mount *mp = context->dp->i_mount;
@@ -89,13 +89,13 @@ xfs_getparent_listent(/* Format the parent pointer directly into the caller buffer. */ ppi->gp_offsets[ppi->gp_count] = context->firstu; pptr =xfs_getparents_rec (ppi, ppi->gp_count); - pptr->xpp_ino = irec->p_ino; - pptr->xpp_gen = irec->p_gen; - pptr->xpp_diroffset = irec->p_diroffset; - pptr->xpp_rsvd =0 ; + pptr->gpr_ino = irec->p_ino; + pptr->gpr_gen = irec->p_gen; + pptr->gpr_diroffset = irec->p_diroffset; + pptr->gpr_rsvd =0 ; -memcpy (pptr->xpp_name, irec->p_name, irec->p_namelen); - pptr->xpp_name[irec->p_namelen] =0 ; +memcpy (pptr->gpr_name, irec->p_name, irec->p_namelen); + pptr->gpr_name[irec->p_namelen] =0 ; ppi->gp_count++; }
PATCH 4/7 xfs: fix GETPARENTS ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix a few remaining issues with this ioctl:
The ioctl encodes the size of the parent rec, not the parent head.
The parent rec should say that it returns a null terminated filename.
The parent head encodes the buffer size, not the size of the parent record array, but the field name and documentation doesn't make this clear.
The getparents sizeof function is pointless and wrong.
Get rid of the last vestiges of the non-flex-array definitions.
The rec address should take an unsigned argument
Whitespace damage
--- fs/xfs/libxfs/xfs_fs.h | 30 +++++++++++------------------- fs/xfs/xfs_ioctl.c | 15 +++++++-------- fs/xfs/xfs_parent_utils.c | 3 +-- 3 files changed, 19 insertions(+), 29 deletions(-)
fs/xfs/libxfs/xfs_fs.h +11 -19
@@ -752,8 +752,6 @@ struct xfs_scrub_metadata { XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED)#define XFS_SCRUB_FLAGS_ALL (XFS_SCRUB_FLAGS_IN | XFS_SCRUB_FLAGS_OUT) -#define XFS_GETPARENTS_MAXNAMELEN 256 -/* return parents of the handle, not the open fd */ #define XFS_GETPARENTS_IFLAG_HANDLE (1U << 0)
@@ -769,11 +767,11 @@ struct xfs_scrub_metadata {/* Get an inode parent pointer through ioctl */ struct xfs_getparents_rec { -__u64 gpr_ino;/* Inode */ -__u32 gpr_gen;/* Inode generation */ -__u32 gpr_diroffset;/* Directory offset */ -__u64 gpr_rsvd;/* Reserved */ -__u8 gpr_name[];/* File name */ +__u64 gpr_ino;/* Inode number */ +__u32 gpr_gen;/* Inode generation */ +__u32 gpr_diroffset;/* Directory offset */ +__u64 gpr_rsvd;/* Reserved */ +__u8 gpr_name[];/* File name and null terminator */ };/* Iterate through an inodes parent pointers */
@@ -794,8 +792,8 @@ struct xfs_getparents {/* Must be set to zero */ __u32 gp_reserved; -/* size of the trailing buffer in bytes */ -__u32 gp_ptrs_size; +/* Size of the buffer in bytes, including this header */ +__u32 gp_bufsize;/* # of entries filled in (output) */ __u32 gp_count;
@@ -807,19 +805,13 @@ struct xfs_getparents {__u32 gp_offsets[]; }; -static inline size_t -xfs_getparents_sizeof (int nr_ptrs) -{ -return sizeof (struct xfs_getparents ) + - (nr_ptrs *sizeof (struct xfs_getparents_rec )); -} -static inline struct xfs_getparents_rec *xfs_getparents_rec (struct xfs_getparents *info, -int idx) +unsigned int idx) { -return (struct xfs_getparents_rec *)((char *)info + info->gp_offsets[idx]); +return (struct xfs_getparents_rec *)((char *)info + + info->gp_offsets[idx]); }/*
@@ -867,7 +859,7 @@ xfs_getparents_rec(/* XFS_IOC_GETFSMAP ------ hoisted 59 */ #define XFS_IOC_SCRUB_METADATA _IOWR('X', 60, struct xfs_scrub_metadata)#define XFS_IOC_AG_GEOMETRY _IOWR('X', 61, struct xfs_ag_geometry) -#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_getparents_rec) +#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_getparents)/* * ioctl commands that replace IRIX syssgi()'s
fs/xfs/xfs_ioctl.c +7 -8
@@ -1681,10 +1681,10 @@ xfs_ioc_scrub_metadata(* IOCTL routine to get the parent pointers of an inode and return it to user * space. Caller must pass a buffer space containing a struct xfs_getparents, * followed by a region large enough to contain an array of struct -* xfs_getparents_rec of a size specified in gp_ptrs_size. If the inode contains +* xfs_getparents_rec of a size specified in gp_bufsize. If the inode contains * more parent pointers than can fit in the buffer space, caller may re-call * the function using the returned gp_cursor to resume iteration. The -* number of xfs_getparents_rec returned will be stored in gp_ptrs_count. +* number of xfs_getparents_rec returned will be stored in gp_count. * * Returns 0 on success or non-zero on failure */
@@ -1699,7 +1699,7 @@ xfs_ioc_get_parent_pointer(struct xfs_inode *call_ip = file_ip;struct xfs_mount *mp = file_ip->i_mount;void __user *o_pptr; -struct xfs_getparents_rec *i_pptr; +struct xfs_getparents_rec *i_pptr;unsigned int bytes;if (!capable (CAP_SYS_ADMIN ))
@@ -1718,11 +1718,11 @@ xfs_ioc_get_parent_pointer( }/* Check size of buffer requested by user */ -if (ppi->gp_ptrs_size >XFS_XATTR_LIST_MAX ) { +if (ppi->gp_bufsize >XFS_XATTR_LIST_MAX ) { error = -ENOMEM ; goto out; } -if (ppi->gp_ptrs_size <sizeof (struct xfs_getparents )) { +if (ppi->gp_bufsize <sizeof (struct xfs_getparents )) { error = -EINVAL ; goto out; }
@@ -1737,8 +1737,7 @@ xfs_ioc_get_parent_pointer(* Now that we know how big the trailing buffer is, expand * our kernel xfs_getparents to be the same size */ - ppi =kvrealloc (ppi,sizeof (struct xfs_getparents ), -xfs_getparents_sizeof (ppi->gp_ptrs_size), + ppi =kvrealloc (ppi,sizeof (struct xfs_getparents ), ppi->gp_bufsize,GFP_KERNEL | __GFP_ZERO);if (!ppi)return -ENOMEM ;
@@ -1794,7 +1793,7 @@ xfs_ioc_get_parent_pointer(/* Copy the parent pointer records back to the user. */ o_pptr = (__user char *)arg + ppi->gp_offsets[ppi->gp_count -1 ]; i_pptr =xfs_getparents_rec (ppi, ppi->gp_count -1 ); - bytes = ((char *)ppi + ppi->gp_ptrs_size) - (char *)i_pptr; + bytes = ((char *)ppi + ppi->gp_bufsize) - (char *)i_pptr; error =copy_to_user (o_pptr, i_pptr, bytes);if (error) { error = -EFAULT ;
fs/xfs/xfs_parent_utils.c +1 -2
@@ -115,7 +115,7 @@ xfs_getparent_pointers( gp->context.dp = ip; gp->context.resynch =1 ; gp->context.put_listent = xfs_getparent_listent; - gp->context.bufsize =round_down (ppi->gp_ptrs_size,sizeof (uint32_t )); + gp->context.bufsize =round_down (ppi->gp_bufsize,sizeof (uint32_t )); gp->context.firstu = gp->context.bufsize;/* Copy the cursor provided by caller */
@@ -149,4 +149,3 @@ xfs_getparent_pointers(kfree (gp);return error; } -
PATCH 5/7 xfs: add tracing to the GETPARENTS ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add some tracing to the getparents ioctl for easier debugging.
--- fs/xfs/xfs_parent_utils.c | 4 ++ fs/xfs/xfs_trace.c | 1 + fs/xfs/xfs_trace.h | 76 ++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 80 insertions(+), 1 deletion(-)
fs/xfs/xfs_parent_utils.c +4 -0
@@ -86,6 +86,8 @@ xfs_getparent_listent(return ; } +trace_xfs_getparent_listent (context->dp, ppi, irec); +/* Format the parent pointer directly into the caller buffer. */ ppi->gp_offsets[ppi->gp_count] = context->firstu; pptr =xfs_getparents_rec (ppi, ppi->gp_count);
@@ -123,6 +125,8 @@ xfs_getparent_pointers( sizeof(struct xfs_attrlist_cursor)); ppi->gp_count =0 ; +trace_xfs_getparent_pointers (ip, ppi, &gp->context.cursor); + error =xfs_attr_list (&gp->context);if (error) goto out_free;
fs/xfs/xfs_trace.c +1 -0
@@ -36,6 +36,7 @@#include "xfs_error.h" #include <linux/iomap.h> #include "xfs_iomap.h" +#include "xfs_parent.h" /* * We include this last to have the helpers above available for the trace
fs/xfs/xfs_trace.h +75 -1
92 lines not on this page · raw message
PATCH 6/7 xfs: shorten parent pointer function names
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Shorten the function names and add brief comments to each, outlining what they're supposed to be doing.
--- fs/xfs/libxfs/xfs_parent.c | 18 ++++++++++++------ fs/xfs/libxfs/xfs_parent.h | 24 ++++++++++++------------ fs/xfs/xfs_inode.c | 16 ++++++++-------- fs/xfs/xfs_symlink.c | 2 +- 4 files changed, 33 insertions(+), 27 deletions(-)
fs/xfs/libxfs/xfs_parent.c +12 -6
@@ -135,6 +135,10 @@ xfs_parent_irec_from_disk(memset (&irec->p_name[valuelen],0 ,sizeof (irec->p_name) - valuelen); } +/* +* Allocate memory to control a logged parent pointer update as part of a +* dirent operation. +*/ int __xfs_parent_init( struct xfs_mount *mp,
@@ -170,12 +174,13 @@ __xfs_parent_init(return 0 ; } +/* Add a parent pointer to reflect a dirent addition. */ int -xfs_parent_defer_add ( +xfs_parent_add (struct xfs_trans *tp,struct xfs_parent_defer *parent,struct xfs_inode *dp, -struct xfs_name *parent_name, +const struct xfs_name *parent_name,xfs_dir2_dataptr_t diroffset,struct xfs_inode *child) {
@@ -194,8 +199,9 @@ xfs_parent_defer_add(return xfs_attr_defer_add (args); } +/* Remove a parent pointer to reflect a dirent removal. */ int -xfs_parent_defer_remove( +xfs_parent_remove( struct xfs_trans *tp, struct xfs_inode *dp, struct xfs_parent_defer *parent,
@@ -211,14 +217,14 @@ xfs_parent_defer_remove(return xfs_attr_defer_remove (args); } - +/* Replace one parent pointer with another to reflect a rename. */ int -xfs_parent_defer_replace ( +xfs_parent_replace (struct xfs_trans *tp,struct xfs_parent_defer *new_parent,struct xfs_inode *old_dp,xfs_dir2_dataptr_t old_diroffset, -struct xfs_name *parent_name, +const struct xfs_name *parent_name,struct xfs_inode *new_dp,xfs_dir2_dataptr_t new_diroffset,struct xfs_inode *child)
fs/xfs/libxfs/xfs_parent.h +12 -12
@@ -49,8 +49,9 @@ struct xfs_parent_defer {* Parent pointer attribute prototypes */ void xfs_init_parent_name_rec (struct xfs_parent_name_rec *rec, -struct xfs_inode *ip, -uint32_t p_diroffset); +struct xfs_inode *ip,uint32_t p_diroffset); +void xfs_init_parent_name_irec (struct xfs_parent_name_irec *irec, +struct xfs_parent_name_rec *rec);int __xfs_parent_init (struct xfs_mount *mp,bool grab_log,struct xfs_parent_defer **parentp);
@@ -78,18 +79,17 @@ xfs_parent_start_locked(return 0 ; } -int xfs_parent_defer_add (struct xfs_trans *tp,struct xfs_parent_defer *parent, -struct xfs_inode *dp,struct xfs_name *parent_name, -xfs_dir2_dataptr_t diroffset,struct xfs_inode *child); -int xfs_parent_defer_replace (struct xfs_trans *tp, +int xfs_parent_add (struct xfs_trans *tp,struct xfs_parent_defer *parent, +struct xfs_inode *dp,const struct xfs_name *parent_name, +xfs_dir2_dataptr_t diroffset,struct xfs_inode *child); +int xfs_parent_replace (struct xfs_trans *tp,struct xfs_parent_defer *new_parent,struct xfs_inode *old_dp, -xfs_dir2_dataptr_t old_diroffset,struct xfs_name *parent_name, -struct xfs_inode *new_ip,xfs_dir2_dataptr_t new_diroffset, +xfs_dir2_dataptr_t old_diroffset, +const struct xfs_name *parent_name,struct xfs_inode *new_ip, +xfs_dir2_dataptr_t new_diroffset,struct xfs_inode *child); +int xfs_parent_remove (struct xfs_trans *tp,struct xfs_inode *dp, +struct xfs_parent_defer *parent,xfs_dir2_dataptr_t diroffset,struct xfs_inode *child); -int xfs_parent_defer_remove (struct xfs_trans *tp,struct xfs_inode *dp, -struct xfs_parent_defer *parent, -xfs_dir2_dataptr_t diroffset, -struct xfs_inode *child);void __xfs_parent_cancel (struct xfs_mount *mp,struct xfs_parent_defer *parent);
fs/xfs/xfs_inode.c +8 -8
@@ -1090,7 +1090,7 @@ xfs_create(* the parent information now. */ if (parent) { - error =xfs_parent_defer_add (tp, parent, dp, name, diroffset, + error =xfs_parent_add (tp, parent, dp, name, diroffset, ip);if (error) goto out_trans_cancel;
@@ -1355,7 +1355,7 @@ xfs_link(* the parent to the inode. */ if (parent) { - error =xfs_parent_defer_add (tp, parent, tdp, target_name, + error =xfs_parent_add (tp, parent, tdp, target_name, diroffset, sip);if (error) goto error_return;
@@ -2626,7 +2626,7 @@ xfs_remove( }if (parent) { - error =xfs_parent_defer_remove (tp, dp, parent, dir_offset, ip); + error =xfs_parent_remove (tp, dp, parent, dir_offset, ip);if (error) goto out_trans_cancel; }
@@ -2835,12 +2835,12 @@ xfs_cross_rename( }if (xfs_has_parent (mp)) { - error =xfs_parent_defer_replace (tp, ip1_pptr, dp1, + error =xfs_parent_replace (tp, ip1_pptr, dp1, old_diroffset, name2, dp2, new_diroffset, ip1);if (error) goto out_trans_abort; - error =xfs_parent_defer_replace (tp, ip2_pptr, dp2, + error =xfs_parent_replace (tp, ip2_pptr, dp2, new_diroffset, name1, dp1, old_diroffset, ip2);if (error) goto out_trans_abort;
@@ -3310,7 +3310,7 @@ xfs_rename( goto out_trans_cancel;if (wip_pptr) { - error =xfs_parent_defer_add (tp, wip_pptr, + error =xfs_parent_add (tp, wip_pptr, src_dp, src_name, old_diroffset, wip);if (error)
@@ -3318,7 +3318,7 @@ xfs_rename( }if (src_ip_pptr) { - error =xfs_parent_defer_replace (tp, src_ip_pptr, src_dp, + error =xfs_parent_replace (tp, src_ip_pptr, src_dp, old_diroffset, target_name, target_dp, new_diroffset, src_ip);if (error)
@@ -3326,7 +3326,7 @@ xfs_rename( }if (tgt_ip_pptr) { - error =xfs_parent_defer_remove (tp, target_dp, + error =xfs_parent_remove (tp, target_dp, tgt_ip_pptr, new_diroffset, target_ip);if (error)
fs/xfs/xfs_symlink.c +1 -1
@@ -348,7 +348,7 @@ xfs_symlink(xfs_trans_log_inode (tp, dp,XFS_ILOG_CORE );if (parent) { - error =xfs_parent_defer_add (tp, parent, dp, link_name, + error =xfs_parent_add (tp, parent, dp, link_name, diroffset, ip);if (error) goto out_trans_cancel;
PATCH 7/7 xfs: rearrange bits of the parent pointer apis for fsck
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Rearrange parts of this thing in preparation for fsck code.
--- fs/xfs/libxfs/xfs_da_format.h | 11 +++++++++++ fs/xfs/libxfs/xfs_parent.c | 29 ++++++++++++++++++++++++++++- fs/xfs/libxfs/xfs_parent.h | 6 ++---- 3 files changed, 41 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_da_format.h +11 -0
@@ -159,6 +159,17 @@ struct xfs_da3_intnode {#define XFS_DIR3_FT_MAX 9 +#define XFS_DIR3_FTYPE_STR \ + { XFS_DIR3_FT_UNKNOWN, "unknown" }, \ + { XFS_DIR3_FT_REG_FILE, "file" }, \ + { XFS_DIR3_FT_DIR, "directory" }, \ + { XFS_DIR3_FT_CHRDEV, "char" }, \ + { XFS_DIR3_FT_BLKDEV, "block" }, \ + { XFS_DIR3_FT_FIFO, "fifo" }, \ + { XFS_DIR3_FT_SOCK, "sock" }, \ + { XFS_DIR3_FT_SYMLINK, "symlink" }, \ + { XFS_DIR3_FT_WHT, "whiteout" } +/* * Byte offset in data block and shortform entry. */
fs/xfs/libxfs/xfs_parent.c +28 -1
@@ -91,7 +91,7 @@ xfs_parent_valuecheck( }/* Initializes a xfs_parent_name_rec to be stored as an attribute name */ -void +static inline void xfs_init_parent_name_rec( struct xfs_parent_name_rec *rec, struct xfs_inode *ip,
@@ -135,6 +135,33 @@ xfs_parent_irec_from_disk(memset (&irec->p_name[valuelen],0 ,sizeof (irec->p_name) - valuelen); } +/* +* Convert an incore parent_name record to its ondisk format. If @value or +* @valuelen are NULL, they will not be written to. +*/ +void +xfs_parent_irec_to_disk ( +struct xfs_parent_name_rec *rec, +void *value, +int *valuelen, +const struct xfs_parent_name_irec *irec) +{ + rec->p_ino =cpu_to_be64 (irec->p_ino); + rec->p_gen =cpu_to_be32 (irec->p_gen); + rec->p_diroffset =cpu_to_be32 (irec->p_diroffset); + +if (valuelen) { +ASSERT (*valuelen >0 ); +ASSERT (*valuelen >= irec->p_namelen); +ASSERT (*valuelen <MAXNAMELEN ); + + *valuelen = irec->p_namelen; + } + +if (value) +memcpy (value, irec->p_name, irec->p_namelen); +} +/* * Allocate memory to control a logged parent pointer update as part of a * dirent operation.
fs/xfs/libxfs/xfs_parent.h +2 -4
@@ -33,6 +33,8 @@ struct xfs_parent_name_irec {void xfs_parent_irec_from_disk (struct xfs_parent_name_irec *irec,const struct xfs_parent_name_rec *rec,const void *value,int valuelen); +void xfs_parent_irec_to_disk (struct xfs_parent_name_rec *rec,void *value, +int *valuelen,const struct xfs_parent_name_irec *irec);/* * Dynamically allocd structure used to wrap the needed data to pass around
@@ -48,10 +50,6 @@ struct xfs_parent_defer {/* * Parent pointer attribute prototypes */ -void xfs_init_parent_name_rec (struct xfs_parent_name_rec *rec, -struct xfs_inode *ip,uint32_t p_diroffset); -void xfs_init_parent_name_irec (struct xfs_parent_name_irec *irec, -struct xfs_parent_name_rec *rec);int __xfs_parent_init (struct xfs_mount *mp,bool grab_log,struct xfs_parent_defer **parentp);
cover xfs: encode parent pointer name in xattr key
In reply to Darrick J. WongHi all,
As I've mentioned in past comments on the parent pointers patchset, the proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:
(parent_ino, parent_gen, diroffset) -> (dirent_name)
If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take a diroffset and try to create the new entry at that offset. This will not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping). Requiring matching diroffsets also prevents reorganization and compaction of directories.
This could be remedied by recording the parent pointer diroffset updates necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new directory structure. If there are a large number of diroffset updates, then the directory commit could take an even longer time.
Worse yet, if the logged xattr updates fill up the transaction, repair will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes. This is a completely bad engineering choice.
Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULL
The xattr code supports xattrs with zero-length values, surprisingly. The parent_gen field makes it easy to export parent handle information, so it can be retained:
(parent_ino, parent_gen, dirent_name) -> NULL
Moving the ondisk format to this format is very advantageous for repair code. Unfortunately, there is one hitch: xattr names cannot exceed 255 bytes due to ondisk format limitations. We don't want to constrain the length of dirent names, so instead we create a special VLOOKUP mode for extended attributes that allows parent pointers to require matching on both the name and the value.
The ondisk format of a parent pointer can then become:
(parent_ino, parent_gen, dirent_name[0:242]) -> (dirent_name[243:255])
Because we can always look up a specific parent pointer. Most of the patches in this patchset prepare the high level xattr code and the lower level logging code to do this correctly, and the last patch switches the ondisk format.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-name-in-attr-key
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-name-in-attr-key
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key
--- fs/xfs/libxfs/xfs_attr.c | 66 +++++--- fs/xfs/libxfs/xfs_attr.h | 5 + fs/xfs/libxfs/xfs_attr_leaf.c | 41 ++++- fs/xfs/libxfs/xfs_da_btree.h | 6 + fs/xfs/libxfs/xfs_da_format.h | 39 ++++- fs/xfs/libxfs/xfs_fs.h | 2 fs/xfs/libxfs/xfs_log_format.h | 31 +++- fs/xfs/libxfs/xfs_parent.c | 215 +++++++++++++++++---------- fs/xfs/libxfs/xfs_parent.h | 46 +++--- fs/xfs/libxfs/xfs_trans_resv.c | 7 + fs/xfs/scrub/dir.c | 34 +--- fs/xfs/scrub/dir_repair.c | 87 +++-------- fs/xfs/scrub/parent.c | 48 +----- fs/xfs/scrub/parent_repair.c | 55 ++----- fs/xfs/scrub/trace.h | 65 +------- fs/xfs/xfs_attr_item.c | 318 +++++++++++++++++++++++++++++++--------- fs/xfs/xfs_attr_item.h | 3 fs/xfs/xfs_inode.c | 30 ++-- fs/xfs/xfs_ondisk.h | 1 fs/xfs/xfs_parent_utils.c | 8 + fs/xfs/xfs_symlink.c | 3 fs/xfs/xfs_xattr.c | 5 + 22 files changed, 660 insertions(+), 455 deletions(-)
PATCH 1/17 xfs: document the ri_total validation in xlog_recover_attri_commit_pass2
From: Darrick J. Wong <[email protected]>
Leave comments explaining why log recovery wants the log item buffer counts to be what they are.
--- fs/xfs/xfs_attr_item.c | 3 +++ 1 file changed, 3 insertions(+)
fs/xfs/xfs_attr_item.c +3 -0
@@ -769,6 +769,7 @@ xlog_recover_attri_commit_pass2(switch (op) {case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE : +/* Log item, attr name, attr value */ if (item->ri_total !=3 ) {XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, attri_formatp, len);
@@ -776,6 +777,7 @@ xlog_recover_attri_commit_pass2( }break ;case XFS_ATTRI_OP_FLAGS_REMOVE : +/* Log item, attr name */ if (item->ri_total !=2 ) {XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, attri_formatp, len);
@@ -783,6 +785,7 @@ xlog_recover_attri_commit_pass2( }break ;case XFS_ATTRI_OP_FLAGS_NVREPLACE : +/* Log item, attr name, new attr name, attr value */ if (item->ri_total !=4 ) {XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, attri_formatp, len);
PATCH 2/17 xfs: make xfs_attr_set require XFS_DA_OP_REMOVE
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Quite a few patches from now, we're going to change the parent pointer xattr format to encode as much of the dirent name in the xattr name as fits, and spill the rest of it to the xattr value. To make this work correctly, we'll be adding the ability to look up xattrs based on name /and/ value.
Internally, the xattr data structure supports attributes with a zero length value, which is how we're going to store parent pointers for short dirent names. The parent pointer repair code uses xfs_attr_set to add missing and remove dangling parent pointers, so that interface must be capable of setting an xattr with args->value == NULL.
The userspace API doesn't support this, so xfs_attr_set currently treats a NULL args->value as a request to remove an attr. However, that's a quirk of the existing callers and the interface. Make the callers of xfs_attr_set to declare explicitly that they want to remove an xattr.
--- fs/xfs/libxfs/xfs_attr.c | 9 +++++---- fs/xfs/libxfs/xfs_parent.c | 1 + fs/xfs/xfs_xattr.c | 5 +++++ 3 files changed, 11 insertions(+), 4 deletions(-)
fs/xfs/libxfs/xfs_attr.c +5 -4
@@ -977,6 +977,7 @@ xfs_attr_set(struct xfs_inode *dp = args->dp;struct xfs_mount *mp = dp->i_mount;struct xfs_trans_res tres; +bool is_remove = args->op_flags &XFS_DA_OP_REMOVE ;bool rsvd;int error, local;int rmt_blks =0 ;
@@ -1004,7 +1005,7 @@ xfs_attr_set( args->op_flags =XFS_DA_OP_OKNOENT | (args->op_flags &XFS_DA_OP_LOGGED ); -if (args->value) { +if (!is_remove) {XFS_STATS_INC (mp, xs_attr_set); args->total =xfs_attr_calc_size (args, &local);
@@ -1038,7 +1039,7 @@ xfs_attr_set(if (error)return error; -if (args->value ||xfs_inode_hasattr (dp)) { +if (!is_remove ||xfs_inode_hasattr (dp)) { error =xfs_iext_count_may_overflow (dp,XFS_ATTR_FORK ,XFS_IEXT_ATTR_MANIP_CNT (rmt_blks));if (error == -EFBIG )
@@ -1052,7 +1053,7 @@ xfs_attr_set(switch (error) {case -EEXIST :/* if no value, we are performing a remove operation */ -if (!args->value) { +if (is_remove) { error =xfs_attr_defer_remove (args);break ; }
@@ -1064,7 +1065,7 @@ xfs_attr_set(break ;case -ENOATTR :/* Can't remove what isn't there. */ -if (!args->value) +if (is_remove) goto out_trans_cancel;/* Pure replace fails if no existing attr to replace. */
fs/xfs/libxfs/xfs_parent.c +1 -0
@@ -384,6 +384,7 @@ xfs_parent_unset( scr->args.geo = ip->i_mount->m_attr_geo; scr->args.name = (const unsigned char *)&scr->rec; scr->args.namelen =sizeof (struct xfs_parent_name_rec ); + scr->args.op_flags =XFS_DA_OP_REMOVE ; scr->args.whichfork =XFS_ATTR_FORK ;return xfs_attr_set (&scr->args);
fs/xfs/xfs_xattr.c +5 -0
@@ -103,6 +103,11 @@ xfs_attr_change( use_logging = true; } +if (args->value) + args->op_flags &= ~XFS_DA_OP_REMOVE ; +else + args->op_flags |=XFS_DA_OP_REMOVE ; + error =xfs_attr_set (args);if (use_logging)
PATCH 3/17 xfs: allow xattr matching on value for local/sf attrs
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add a new VLOOKUP flag to signal that the caller wants to look up an extended attribute by name and value. This only works with shortform and local attributes. Only parent pointers need this functionality, parent pointers cannot be remote xattrs, hence this limitation is ok.
--- fs/xfs/libxfs/xfs_attr_leaf.c | 41 +++++++++++++++++++++++++++++++++++------ fs/xfs/libxfs/xfs_da_btree.h | 4 +++- 2 files changed, 38 insertions(+), 7 deletions(-)
fs/xfs/libxfs/xfs_attr_leaf.c +35 -6
@@ -473,10 +473,12 @@ xfs_attr3_leaf_read(*/ static bool xfs_attr_match ( -struct xfs_da_args *args, -uint8_t namelen, -unsigned char *name, -int flags) +const struct xfs_da_args *args, +uint8_t namelen, +const unsigned char *name, +unsigned int valuelen, +const void *value, +int flags) {if (args->namelen != namelen)
@@ -484,6 +486,23 @@ xfs_attr_match(if (memcmp (args->name, name, namelen) !=0 )return false; +if (args->op_flags &XFS_DA_OP_VLOOKUP ) { +if (args->valuelen != valuelen) +return false; +if (args->valuelen && !value) { +/* not implemented for remote values */ +ASSERT (0 ); +return false; + } +if (valuelen && !args->value) { +/* caller gave us valuelen > 0 but no value?? */ +ASSERT (0 ); +return false; + } +if (valuelen >0 &&memcmp (args->value, value, valuelen) !=0 ) +return false; + } +/* Recovery ignores the INCOMPLETE flag. */ if ((args->op_flags &XFS_DA_OP_RECOVERY ) && args->attr_filter == (flags &XFS_ATTR_NSP_ONDISK_MASK ))
@@ -502,6 +521,10 @@ xfs_attr_copy_value( unsigned char *value, int valuelen) { +/* vlookups already supplied the attr value; don't copy anything */ +if (args->op_flags &XFS_DA_OP_VLOOKUP ) +return 0 ; +/* * No copy if all we have to do is get the length */
@@ -726,6 +749,7 @@ xfs_attr_sf_findname( base += size, i++) { size =xfs_attr_sf_entsize (sfe);if (!xfs_attr_match (args, sfe->namelen, sfe->nameval, + sfe->valuelen, &sfe->nameval[sfe->namelen], sfe->flags))continue ;break ;
@@ -896,6 +920,7 @@ xfs_attr_shortform_lookup(xfs_da_args_t *args)for (i =0 ; i < sf->hdr.count; sfe =xfs_attr_sf_nextentry (sfe), i++) {if (xfs_attr_match (args, sfe->namelen, sfe->nameval, + sfe->valuelen, &sfe->nameval[sfe->namelen], sfe->flags))return -EEXIST ; }
@@ -923,6 +948,7 @@ xfs_attr_shortform_getvalue(for (i =0 ; i < sf->hdr.count; sfe =xfs_attr_sf_nextentry (sfe), i++) {if (xfs_attr_match (args, sfe->namelen, sfe->nameval, + sfe->valuelen, &sfe->nameval[sfe->namelen], sfe->flags))return xfs_attr_copy_value (args, &sfe->nameval[args->namelen], sfe->valuelen);
@@ -2484,14 +2510,17 @@ xfs_attr3_leaf_lookup_int(if (entry->flags &XFS_ATTR_LOCAL ) { name_loc =xfs_attr3_leaf_name_local (leaf, probe);if (!xfs_attr_match (args, name_loc->namelen, - name_loc->nameval, entry->flags)) + name_loc->nameval, +be16_to_cpu (name_loc->valuelen), + &name_loc->nameval[name_loc->namelen], + entry->flags))continue ; args->index = probe;return -EEXIST ; }else { name_rmt =xfs_attr3_leaf_name_remote (leaf, probe);if (!xfs_attr_match (args, name_rmt->namelen, - name_rmt->name, entry->flags)) + name_rmt->name,0 ,NULL , entry->flags))continue ; args->index = probe; args->rmtvaluelen =be32_to_cpu (name_rmt->valuelen);
fs/xfs/libxfs/xfs_da_btree.h +3 -1
@@ -96,6 +96,7 @@ typedef struct xfs_da_args {#define XFS_DA_OP_REMOVE (1u << 6)/* this is a remove operation */ #define XFS_DA_OP_RECOVERY (1u << 7)/* Log recovery operation */ #define XFS_DA_OP_LOGGED (1u << 8)/* Use intent items to track op */ +#define XFS_DA_OP_VLOOKUP (1u << 9)/* Compare local attr value for lookup */ #define XFS_DA_OP_FLAGS \ { XFS_DA_OP_JUSTCHECK, "JUSTCHECK" }, \
@@ -106,7 +107,8 @@ typedef struct xfs_da_args { { XFS_DA_OP_NOTIME, "NOTIME" }, \ { XFS_DA_OP_REMOVE, "REMOVE" }, \ { XFS_DA_OP_RECOVERY, "RECOVERY" }, \ - { XFS_DA_OP_LOGGED, "LOGGED" } + { XFS_DA_OP_LOGGED, "LOGGED" }, \ + { XFS_DA_OP_VLOOKUP, "VLOOKUP" }/* * Storage for holding state during Btree searches and split/join ops.
PATCH 4/17 xfs: preserve VLOOKUP in xfs_attr_set
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Preserve the attr-value lookup flag when calling xfs_attr_set. Normal xattr users will never use this, but parent pointer fsck will.
--- fs/xfs/libxfs/xfs_attr.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
fs/xfs/libxfs/xfs_attr.c +4 -4
@@ -999,11 +999,11 @@ xfs_attr_set(/* * We have no control over the attribute names that userspace passes us * to remove, so we have to allow the name lookup prior to attribute -* removal to fail as well. Preserve the logged flag, since we need -* to pass that through to the logging code. +* removal to fail as well. Preserve the logged and vlookup flags, +* since we need to pass them through to the lower levels. */ - args->op_flags =XFS_DA_OP_OKNOENT | - (args->op_flags &XFS_DA_OP_LOGGED ); + args->op_flags &= (XFS_DA_OP_LOGGED |XFS_DA_OP_VLOOKUP ); + args->op_flags |=XFS_DA_OP_OKNOENT ;if (!is_remove) {XFS_STATS_INC (mp, xs_attr_set);
PATCH 5/17 xfs: restructure xfs_attr_complete_op a bit
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Reduce the indentation in this function by flattening the nested if statements. We're going to add more code later to this function later, hence the early cleanup. No functional changes.
--- fs/xfs/libxfs/xfs_attr.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
fs/xfs/libxfs/xfs_attr.c +10 -10
@@ -422,17 +422,17 @@ xfs_attr_complete_op(bool do_replace = args->op_flags &XFS_DA_OP_REPLACE ; args->op_flags &= ~XFS_DA_OP_REPLACE ; -if (do_replace) { - args->attr_filter &= ~XFS_ATTR_INCOMPLETE ; -if (args->new_namelen >0 ) { - args->name = args->new_name; - args->namelen = args->new_namelen; - args->hashval =xfs_da_hashname (args->name, - args->namelen); - } +if (!do_replace) +return XFS_DAS_DONE ; + + args->attr_filter &= ~XFS_ATTR_INCOMPLETE ; +if (args->new_namelen ==0 )return replace_state; - } -return XFS_DAS_DONE ; + + args->name = args->new_name; + args->namelen = args->new_namelen; + args->hashval =xfs_da_hashname (args->name, args->namelen); +return replace_state; } static int
PATCH 6/17 xfs: use helpers to extract xattr op from opflags
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create helper functions to extract the xattr op from the ondisk xattri log item and the incore attr intent item. These will get more use in the patches that follow.
--- fs/xfs/libxfs/xfs_attr.h | 5 +++++ fs/xfs/xfs_attr_item.c | 18 +++++++++++------- 2 files changed, 16 insertions(+), 7 deletions(-)
fs/xfs/libxfs/xfs_attr.h +5 -0
@@ -530,6 +530,11 @@ struct xfs_attr_intent {struct xfs_bmbt_irec xattri_map; }; +static inline unsigned int +xfs_attr_intent_op (const struct xfs_attr_intent *attr) +{ +return attr->xattri_op_flags &XFS_ATTRI_OP_FLAGS_TYPE_MASK ; +}/*======================================================================== * Function prototypes for the kernel.
fs/xfs/xfs_attr_item.c +11 -7
@@ -374,6 +374,12 @@ xfs_xattri_finish_update(return error; } +static inline unsigned int +xfs_attr_log_item_op (const struct xfs_attri_log_format *attrp) +{ +return attrp->alfi_op_flags &XFS_ATTRI_OP_FLAGS_TYPE_MASK ; +} +/* Log an attr to the intent item. */ STATIC void xfs_attr_log_item(
@@ -525,10 +531,9 @@ xfs_attri_validate(struct xfs_mount *mp, struct xfs_attri_log_format *attrp) { - unsigned int op = attrp->alfi_op_flags & -XFS_ATTRI_OP_FLAGS_TYPE_MASK ; + unsigned int op = xfs_attr_log_item_op(attrp); -if (attrp->alfi_op_flags !=XFS_ATTRI_OP_FLAGS_NVREPLACE && +if (op !=XFS_ATTRI_OP_FLAGS_NVREPLACE && attrp->alfi_nname_len !=0 )return false;
@@ -606,8 +611,7 @@ xfs_attri_item_recover( args = (struct xfs_da_args *)(attr +1 ); attr->xattri_da_args = args; - attr->xattri_op_flags = attrp->alfi_op_flags & -XFS_ATTRI_OP_FLAGS_TYPE_MASK ; + attr->xattri_op_flags =xfs_attr_log_item_op (attrp);/* * We're reconstructing the deferred work state structure from the
@@ -631,7 +635,7 @@ xfs_attri_item_recover(ASSERT (xfs_sb_version_haslogxattrs (&mp->m_sb)); -switch (attr->xattri_op_flags) { +switch (xfs_attr_intent_op (attr)) {case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :case XFS_ATTRI_OP_FLAGS_NVREPLACE :
@@ -765,7 +769,7 @@ xlog_recover_attri_commit_pass2(return -EFSCORRUPTED ; } - op = attri_formatp->alfi_op_flags &XFS_ATTRI_OP_FLAGS_TYPE_MASK ; + op =xfs_attr_log_item_op (attri_formatp);switch (op) {case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :
PATCH 7/17 xfs: validate recovered name buffers when recovering xattr items
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Strengthen the xattri log item recovery code by checking that we actually have the required name and newname buffers for whatever operation we're replaying.
--- fs/xfs/xfs_attr_item.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 59 insertions(+), 3 deletions(-)
fs/xfs/xfs_attr_item.c +59 -3
@@ -801,8 +801,8 @@ xlog_recover_attri_commit_pass2( attri_formatp, len);return -EFSCORRUPTED ; } - i++; +/* Validate the attr name */ if (item->ri_buf[i].i_len !=xlog_calc_iovec_len (attri_formatp->alfi_name_len)) {
@@ -818,10 +818,10 @@ xlog_recover_attri_commit_pass2( item->ri_buf[i].i_addr, item->ri_buf[i].i_len);return -EFSCORRUPTED ; } - i++; + +/* Validate the attr name. */ if (attri_formatp->alfi_nname_len) { -/* Validate the attr nname */ if (item->ri_buf[i].i_len !=xlog_calc_iovec_len (attri_formatp->alfi_nname_len)) { XFS_CORRUPTION_ERROR(__func__, XFS_ERRLEVEL_LOW, mp,
@@ -852,6 +852,62 @@ xlog_recover_attri_commit_pass2( } attr_value = item->ri_buf[i].i_addr; + i++; + } + +/* +* Make sure we got the correct number of buffers for the operation +* that we just loaded. +*/ +if (i != item->ri_total) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } + +switch (op) { +case XFS_ATTRI_OP_FLAGS_REMOVE : +/* Regular remove operations operate only on names. */ +if (attr_value !=NULL || attri_formatp->alfi_value_len !=0 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } + fallthrough; +case XFS_ATTRI_OP_FLAGS_SET : +case XFS_ATTRI_OP_FLAGS_REPLACE : +/* +* Regular xattr set/remove/replace operations require a name +* and do not take a newname. Values are optional for set and +* replace. +*/ +if (attr_name ==NULL || attri_formatp->alfi_name_len ==0 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } +if (attr_nname !=NULL || attri_formatp->alfi_nname_len !=0 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } +break ; +case XFS_ATTRI_OP_FLAGS_NVREPLACE : +/* +* Name-value replace operations require the caller to specify +* the old and new name explicitly. Values are optional. +*/ +if (attr_name ==NULL || attri_formatp->alfi_name_len ==0 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } +if (attr_nname ==NULL || attri_formatp->alfi_nname_len ==0 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } +break ; }/*
PATCH 8/17 xfs: always set args->value in xfs_attri_item_recover
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Always set args->value to the recovered value buffer. This reduces the amount of code in the switch statement, and hence the amount of thinking that I have to do. We validated the recovered buffers, supposedly.
--- fs/xfs/xfs_attr_item.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/xfs/xfs_attr_item.c +2 -2
@@ -629,6 +629,8 @@ xfs_attri_item_recover( args->new_name = nv->nname.i_addr; args->new_namelen = nv->nname.i_len; args->hashval =xfs_da_hashname (args->name, args->namelen); + args->value = nv->value.i_addr; + args->valuelen = nv->value.i_len; args->attr_filter = attrp->alfi_attr_filter &XFS_ATTRI_FILTER_MASK ; args->op_flags =XFS_DA_OP_RECOVERY |XFS_DA_OP_OKNOENT |XFS_DA_OP_LOGGED ;
@@ -639,8 +641,6 @@ xfs_attri_item_recover(case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :case XFS_ATTRI_OP_FLAGS_NVREPLACE : - args->value = nv->value.i_addr; - args->valuelen = nv->value.i_len; args->total =xfs_attr_calc_size (args, &local);if (xfs_inode_hasattr (args->dp)) attr->xattri_dela_state =xfs_attr_init_replace_state (args);
PATCH 9/17 xfs: flip nvreplace detection in xfs_attr_complete_op
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Gate the NVREPLACE code on the op flags directly, instead of inferring it through args->new_namelen.
--- fs/xfs/libxfs/xfs_attr.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_attr.c +7 -1
@@ -426,9 +426,15 @@ xfs_attr_complete_op(return XFS_DAS_DONE ; args->attr_filter &= ~XFS_ATTR_INCOMPLETE ; -if (args->new_namelen ==0 ) +if (xfs_attr_intent_op (attr) !=XFS_ATTRI_OP_FLAGS_NVREPLACE )return replace_state; +/* +* NVREPLACE operations require the caller to set the old and new names +* explicitly. +*/ +ASSERT (args->new_namelen >0 ); + args->name = args->new_name; args->namelen = args->new_namelen; args->hashval =xfs_da_hashname (args->name, args->namelen);
PATCH 10/17 xfs: log VLOOKUP xattr removal operations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If high level code wants to do a deferred xattr remove operation with the VLOOKUP flag set, we need to push this through the log.
--- fs/xfs/libxfs/xfs_attr.c | 6 +++++- fs/xfs/libxfs/xfs_log_format.h | 1 + fs/xfs/xfs_attr_item.c | 16 ++++++++++++++++ 3 files changed, 22 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_attr.c +5 -1
@@ -959,9 +959,13 @@ xfs_attr_defer_remove( {struct xfs_attr_intent *new; +int op_flag =XFS_ATTRI_OP_FLAGS_REMOVE ;int error; - error =xfs_attr_intent_init (args,XFS_ATTRI_OP_FLAGS_REMOVE , &new); +if (args->op_flags &XFS_DA_OP_VLOOKUP ) + op_flag =XFS_ATTRI_OP_FLAGS_NVREMOVE ; + + error =xfs_attr_intent_init (args, op_flag, &new);if (error)return error;
fs/xfs/libxfs/xfs_log_format.h +1 -0
@@ -959,6 +959,7 @@ struct xfs_icreate_log {#define XFS_ATTRI_OP_FLAGS_REMOVE 2/* Remove the attribute */ #define XFS_ATTRI_OP_FLAGS_REPLACE 3/* Replace the attribute */ #define XFS_ATTRI_OP_FLAGS_NVREPLACE 4/* Replace attr name and val */ +#define XFS_ATTRI_OP_FLAGS_NVREMOVE 5/* Remove attr w/ vlookup */ #define XFS_ATTRI_OP_FLAGS_TYPE_MASK 0xFF/* Flags type mask */ /*
fs/xfs/xfs_attr_item.c +16 -0
@@ -549,6 +549,7 @@ xfs_attri_validate(case XFS_ATTRI_OP_FLAGS_REPLACE :case XFS_ATTRI_OP_FLAGS_REMOVE :case XFS_ATTRI_OP_FLAGS_NVREPLACE : +case XFS_ATTRI_OP_FLAGS_NVREMOVE :break ;default :return false;
@@ -647,6 +648,9 @@ xfs_attri_item_recover(else attr->xattri_dela_state =xfs_attr_init_add_state (args);break ; +case XFS_ATTRI_OP_FLAGS_NVREMOVE : + args->op_flags |=XFS_DA_OP_VLOOKUP ; + fallthrough;case XFS_ATTRI_OP_FLAGS_REMOVE :if (!xfs_inode_hasattr (args->dp)) goto out;
@@ -771,6 +775,14 @@ xlog_recover_attri_commit_pass2( op =xfs_attr_log_item_op (attri_formatp);switch (op) { +case XFS_ATTRI_OP_FLAGS_NVREMOVE : +/* Log item, attr name, optional attr value */ +if (item->ri_total !=3 && item->ri_total !=2 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } +break ;case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :/* Log item, attr name, attr value */
@@ -874,12 +886,16 @@ xlog_recover_attri_commit_pass2(return -EFSCORRUPTED ; } fallthrough; +case XFS_ATTRI_OP_FLAGS_NVREMOVE :case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :/* * Regular xattr set/remove/replace operations require a name * and do not take a newname. Values are optional for set and * replace. +* +* Name-value remove operations must have a name, do not +* take a newname, and can take a value. */ if (attr_name ==NULL || attri_formatp->alfi_name_len ==0 ) { XFS_CORRUPTION_ERROR(__func__, XFS_ERRLEVEL_LOW, mp,
PATCH 11/17 xfs: log VLOOKUP xattr setting operations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If high level code wants to do a deferred xattr set operation with the VLOOKUP flag set, we need to push this through the log.
--- fs/xfs/libxfs/xfs_attr.c | 6 +++++- fs/xfs/libxfs/xfs_log_format.h | 1 + fs/xfs/xfs_attr_item.c | 8 +++++++- 3 files changed, 13 insertions(+), 2 deletions(-)
fs/xfs/libxfs/xfs_attr.c +5 -1
@@ -916,9 +916,13 @@ xfs_attr_defer_add( struct xfs_da_args *args) {struct xfs_attr_intent *new; +int op_flag =XFS_ATTRI_OP_FLAGS_SET ;int error =0 ; - error =xfs_attr_intent_init (args,XFS_ATTRI_OP_FLAGS_SET , &new); +if (args->op_flags &XFS_DA_OP_VLOOKUP ) + op_flag =XFS_ATTRI_OP_FLAGS_NVSET ; + + error =xfs_attr_intent_init (args, op_flag, &new);if (error)return error;
fs/xfs/libxfs/xfs_log_format.h +1 -0
@@ -960,6 +960,7 @@ struct xfs_icreate_log {#define XFS_ATTRI_OP_FLAGS_REPLACE 3/* Replace the attribute */ #define XFS_ATTRI_OP_FLAGS_NVREPLACE 4/* Replace attr name and val */ #define XFS_ATTRI_OP_FLAGS_NVREMOVE 5/* Remove attr w/ vlookup */ +#define XFS_ATTRI_OP_FLAGS_NVSET 6/* Set attr with w/ vlookup */ #define XFS_ATTRI_OP_FLAGS_TYPE_MASK 0xFF/* Flags type mask */ /*
fs/xfs/xfs_attr_item.c +7 -1
@@ -550,6 +550,7 @@ xfs_attri_validate(case XFS_ATTRI_OP_FLAGS_REMOVE :case XFS_ATTRI_OP_FLAGS_NVREPLACE :case XFS_ATTRI_OP_FLAGS_NVREMOVE : +case XFS_ATTRI_OP_FLAGS_NVSET :break ;default :return false;
@@ -639,6 +640,9 @@ xfs_attri_item_recover(ASSERT (xfs_sb_version_haslogxattrs (&mp->m_sb));switch (xfs_attr_intent_op (attr)) { +case XFS_ATTRI_OP_FLAGS_NVSET : + args->op_flags |=XFS_DA_OP_VLOOKUP ; + fallthrough;case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :case XFS_ATTRI_OP_FLAGS_NVREPLACE :
@@ -775,6 +779,7 @@ xlog_recover_attri_commit_pass2( op =xfs_attr_log_item_op (attri_formatp);switch (op) { +case XFS_ATTRI_OP_FLAGS_NVSET :case XFS_ATTRI_OP_FLAGS_NVREMOVE :/* Log item, attr name, optional attr value */ if (item->ri_total !=3 && item->ri_total !=2 ) {
@@ -886,6 +891,7 @@ xlog_recover_attri_commit_pass2(return -EFSCORRUPTED ; } fallthrough; +case XFS_ATTRI_OP_FLAGS_NVSET :case XFS_ATTRI_OP_FLAGS_NVREMOVE :case XFS_ATTRI_OP_FLAGS_SET :case XFS_ATTRI_OP_FLAGS_REPLACE :
@@ -894,7 +900,7 @@ xlog_recover_attri_commit_pass2(* and do not take a newname. Values are optional for set and * replace. * -* Name-value remove operations must have a name, do not +* Name-value set/remove operations must have a name, do not * take a newname, and can take a value. */ if (attr_name ==NULL || attri_formatp->alfi_name_len ==0 ) {
PATCH 12/17 xfs: overlay alfi_nname_len atop alfi_name_len for NVREPLACE
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
In preparation for being able to log the old attr value in a NVREPLACE operation, encode the old and new name lengths in the alfi_name_len field. We haven't shipped a kernel with XFS_ATTRI_OP_FLAGS_NVREPLACE, so we can still tweak the ondisk log item format.
--- fs/xfs/libxfs/xfs_log_format.h | 14 +++++- fs/xfs/xfs_attr_item.c | 99 ++++++++++++++++++++++------------------ 2 files changed, 67 insertions(+), 46 deletions(-)
fs/xfs/libxfs/xfs_log_format.h +12 -2
@@ -979,11 +979,21 @@ struct xfs_icreate_log {struct xfs_attri_log_format {uint16_t alfi_type;/* attri log item type */ uint16_t alfi_size;/* size of this item */ -uint32_t alfi_nname_len;/* attr new name length */ +uint32_t __pad;/* pad to 64 bit aligned */ uint64_t alfi_id;/* attri identifier */ uint64_t alfi_ino;/* the inode for this attr operation */ uint32_t alfi_op_flags;/* marks the op as a set or remove */ -uint32_t alfi_name_len;/* attr name length */ +union { +uint32_t alfi_name_len;/* attr name length */ +struct { +/* +* For NVREPLACE, these are the lengths of the old and +* new attr name. +*/ +uint16_t alfi_oldname_len; +uint16_t alfi_newname_len; + }; + };uint32_t alfi_value_len;/* attr value length */ uint32_t alfi_attr_filter;/* attr filter flags */ };
fs/xfs/xfs_attr_item.c +55 -44
208 lines not on this page · raw message
PATCH 13/17 xfs: refactor value length in xlog_recover_attri_commit_pass2
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use a shorter convenience variable for the attr value length in this function, since we've just done that for attr name lengths.
--- fs/xfs/xfs_attr_item.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
fs/xfs/xfs_attr_item.c +6 -5
@@ -775,6 +775,7 @@ xlog_recover_attri_commit_pass2(size_t len;const void *attr_newname =NULL ;unsigned int name_len =0 , newname_len =0 ; +unsigned int value_len;int op, i =0 ;/* Validate xfs_attri_log_format before the large memory allocation */
@@ -792,6 +793,7 @@ xlog_recover_attri_commit_pass2(return -EFSCORRUPTED ; } + value_len = attri_formatp->alfi_value_len; op =xfs_attr_log_item_op (attri_formatp);switch (op) {case XFS_ATTRI_OP_FLAGS_NVSET :
@@ -878,8 +880,8 @@ xlog_recover_attri_commit_pass2(/* Validate the attr value, if present */ -if (attri_formatp->alfi_value_len !=0 ) { -if (item->ri_buf[i].i_len !=xlog_calc_iovec_len (attri_formatp->alfi_value_len)) { +if (value_len !=0 ) { +if (item->ri_buf[i].i_len !=xlog_calc_iovec_len (value_len)) {XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, attri_formatp, len);return -EFSCORRUPTED ;
@@ -902,7 +904,7 @@ xlog_recover_attri_commit_pass2(switch (op) {case XFS_ATTRI_OP_FLAGS_REMOVE :/* Regular remove operations operate only on names. */ -if (attr_value !=NULL || attri_formatp->alfi_value_len !=0 ) { +if (attr_value !=NULL || value_len !=0 ) {XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, attri_formatp, len);return -EFSCORRUPTED ;
@@ -950,8 +952,7 @@ xlog_recover_attri_commit_pass2(* reference. */ nv =xfs_attri_log_nameval_alloc (attr_name, name_len, attr_newname, - newname_len, attr_value, - attri_formatp->alfi_value_len); + newname_len, attr_value, value_len); attrip =xfs_attri_init (mp, nv);memcpy (&attrip->attri_format, attri_formatp, len);
PATCH 14/17 xfs: rename nname to newname
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Burn a couple of extra source code bytes to make it clear what this does.
--- fs/xfs/libxfs/xfs_log_format.h | 2 +- fs/xfs/xfs_attr_item.c | 40 ++++++++++++++++++++-------------------- fs/xfs/xfs_attr_item.h | 2 +- 3 files changed, 22 insertions(+), 22 deletions(-)
fs/xfs/libxfs/xfs_log_format.h +1 -1
@@ -117,7 +117,7 @@ struct xfs_unmount_log_format {#define XLOG_REG_TYPE_ATTRD_FORMAT 28#define XLOG_REG_TYPE_ATTR_NAME 29#define XLOG_REG_TYPE_ATTR_VALUE 30 -#define XLOG_REG_TYPE_ATTR_NNAME 31 +#define XLOG_REG_TYPE_ATTR_NEWNAME 31#define XLOG_REG_TYPE_MAX 31
fs/xfs/xfs_attr_item.c +20 -20
@@ -75,8 +75,8 @@ static inline struct xfs_attri_log_nameval *xfs_attri_log_nameval_alloc ( const void *name, unsigned int name_len, - const void *nname, - unsigned int nname_len, + const void *newname, + unsigned int newname_len, const void *value, unsigned int value_len) {
@@ -87,25 +87,25 @@ xfs_attri_log_nameval_alloc(* this. But kvmalloc() utterly sucks, so we use our own version. */ nv =xlog_kvmalloc (sizeof (struct xfs_attri_log_nameval ) + - name_len + nname_len + value_len); + name_len + newname_len + value_len); nv->name.i_addr = nv +1 ; nv->name.i_len = name_len; nv->name.i_type =XLOG_REG_TYPE_ATTR_NAME ;memcpy (nv->name.i_addr, name, name_len); -if (nname_len) { - nv->nname.i_addr = nv->name.i_addr + name_len; - nv->nname.i_len = nname_len; -memcpy (nv->nname.i_addr, nname, nname_len); +if (newname_len) { + nv->newname.i_addr = nv->name.i_addr + name_len; + nv->newname.i_len = newname_len; +memcpy (nv->newname.i_addr, newname, newname_len); }else { - nv->nname.i_addr =NULL ; - nv->nname.i_len =0 ; + nv->newname.i_addr =NULL ; + nv->newname.i_len =0 ; } - nv->nname.i_type =XLOG_REG_TYPE_ATTR_NNAME ; + nv->newname.i_type =XLOG_REG_TYPE_ATTR_NEWNAME ;if (value_len) { - nv->value.i_addr = nv->name.i_addr + nname_len + name_len; + nv->value.i_addr = nv->name.i_addr + newname_len + name_len; nv->value.i_len = value_len;memcpy (nv->value.i_addr, value, value_len); }else {
@@ -159,9 +159,9 @@ xfs_attri_item_size( *nbytes +=sizeof (struct xfs_attri_log_format ) +xlog_calc_iovec_len (nv->name.i_len); -if (nv->nname.i_len) { +if (nv->newname.i_len) { *nvecs +=1 ; - *nbytes +=xlog_calc_iovec_len (nv->nname.i_len); + *nbytes +=xlog_calc_iovec_len (nv->newname.i_len); }if (nv->value.i_len) {
@@ -197,7 +197,7 @@ xfs_attri_item_format(ASSERT (nv->name.i_len >0 ); attrip->attri_format.alfi_size++; -if (nv->nname.i_len >0 ) +if (nv->newname.i_len >0 ) attrip->attri_format.alfi_size++;if (nv->value.i_len >0 )
@@ -208,8 +208,8 @@ xfs_attri_item_format( sizeof(struct xfs_attri_log_format));xlog_copy_from_iovec (lv, &vecp, &nv->name); -if (nv->nname.i_len >0 ) -xlog_copy_from_iovec (lv, &vecp, &nv->nname); +if (nv->newname.i_len >0 ) +xlog_copy_from_iovec (lv, &vecp, &nv->newname);if (nv->value.i_len >0 )xlog_copy_from_iovec (lv, &vecp, &nv->value);
@@ -405,7 +405,7 @@ xfs_attr_log_item(if (xfs_attr_log_item_op (attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACE ) { attrp->alfi_oldname_len = attr->xattri_nameval->name.i_len; - attrp->alfi_newname_len = attr->xattri_nameval->nname.i_len; + attrp->alfi_newname_len = attr->xattri_nameval->newname.i_len; }else { attrp->alfi_name_len = attr->xattri_nameval->name.i_len; }
@@ -638,8 +638,8 @@ xfs_attri_item_recover( args->whichfork =XFS_ATTR_FORK ; args->name = nv->name.i_addr; args->namelen = nv->name.i_len; - args->new_name = nv->nname.i_addr; - args->new_namelen = nv->nname.i_len; + args->new_name = nv->newname.i_addr; + args->new_namelen = nv->newname.i_len; args->hashval =xfs_da_hashname (args->name, args->namelen); args->value = nv->value.i_addr; args->valuelen = nv->value.i_len;
@@ -858,7 +858,7 @@ xlog_recover_attri_commit_pass2( } i++; -/* Validate the attr nname */ +/* Validate the new attr name */ if (newname_len >0 ) {if (item->ri_buf[i].i_len !=xlog_calc_iovec_len (newname_len)) { XFS_CORRUPTION_ERROR(__func__, XFS_ERRLEVEL_LOW, mp,
fs/xfs/xfs_attr_item.h +1 -1
@@ -13,7 +13,7 @@ struct kmem_zone;struct xfs_attri_log_nameval {struct xfs_log_iovec name; -struct xfs_log_iovec nname; +struct xfs_log_iovec newname;struct xfs_log_iovec value;refcount_t refcount;
PATCH 15/17 xfs: log VLOOKUP xattr nvreplace operations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If high level code wants to do a deferred xattr nvreplace operation with the VLOOKUP flag set, we need to push this through the log. To avoid breaking bisection in parent pointer code, we'll temporarily create a new NVREPLACEXXX flag that connects to the VLOOKUP flag.
--- fs/xfs/libxfs/xfs_attr.c | 11 +++++++---- fs/xfs/libxfs/xfs_log_format.h | 1 + fs/xfs/xfs_attr_item.c | 23 +++++++++++++++++++++-- 3 files changed, 29 insertions(+), 6 deletions(-)
fs/xfs/libxfs/xfs_attr.c +7 -4
@@ -426,7 +426,8 @@ xfs_attr_complete_op(return XFS_DAS_DONE ; args->attr_filter &= ~XFS_ATTR_INCOMPLETE ; -if (xfs_attr_intent_op (attr) !=XFS_ATTRI_OP_FLAGS_NVREPLACE ) +if (xfs_attr_intent_op (attr) !=XFS_ATTRI_OP_FLAGS_NVREPLACE && +xfs_attr_intent_op (attr) !=XFS_ATTRI_OP_FLAGS_NVREPLACEXXX )return replace_state;/*
@@ -939,11 +940,13 @@ xfs_attr_defer_replace( struct xfs_da_args *args) {struct xfs_attr_intent *new; -int op_flag; +int op_flag =XFS_ATTRI_OP_FLAGS_REPLACE ;int error =0 ; - op_flag = args->new_namelen ==0 ?XFS_ATTRI_OP_FLAGS_REPLACE : -XFS_ATTRI_OP_FLAGS_NVREPLACE ; +if (args->op_flags &XFS_DA_OP_VLOOKUP ) + op_flag =XFS_ATTRI_OP_FLAGS_NVREPLACEXXX ; +else if (args->new_namelen >0 ) + op_flag =XFS_ATTRI_OP_FLAGS_NVREPLACE ; error =xfs_attr_intent_init (args, op_flag, &new);if (error)
fs/xfs/libxfs/xfs_log_format.h +1 -0
@@ -961,6 +961,7 @@ struct xfs_icreate_log {#define XFS_ATTRI_OP_FLAGS_NVREPLACE 4/* Replace attr name and val */ #define XFS_ATTRI_OP_FLAGS_NVREMOVE 5/* Remove attr w/ vlookup */ #define XFS_ATTRI_OP_FLAGS_NVSET 6/* Set attr with w/ vlookup */ +#define XFS_ATTRI_OP_FLAGS_NVREPLACEXXX 7/* Replace attr name and val */ #define XFS_ATTRI_OP_FLAGS_TYPE_MASK 0xFF/* Flags type mask */ /*
fs/xfs/xfs_attr_item.c +21 -2
@@ -403,7 +403,10 @@ xfs_attr_log_item( attrp->alfi_op_flags = attr->xattri_op_flags; attrp->alfi_value_len = attr->xattri_nameval->value.i_len; -if (xfs_attr_log_item_op (attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACE ) { +if (xfs_attr_log_item_op (attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACEXXX ) { + attrp->alfi_oldname_len = attr->xattri_nameval->name.i_len; + attrp->alfi_newname_len = attr->xattri_nameval->newname.i_len; + }else if (xfs_attr_log_item_op (attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACE ) { attrp->alfi_oldname_len = attr->xattri_nameval->name.i_len; attrp->alfi_newname_len = attr->xattri_nameval->newname.i_len; }else {
@@ -564,6 +567,7 @@ xfs_attri_validate(if (attrp->alfi_value_len >XATTR_SIZE_MAX )return false;break ; +case XFS_ATTRI_OP_FLAGS_NVREPLACEXXX :case XFS_ATTRI_OP_FLAGS_NVREPLACE :if (attrp->alfi_oldname_len ==0 || attrp->alfi_oldname_len >XATTR_NAME_MAX )
@@ -650,6 +654,7 @@ xfs_attri_item_recover(ASSERT (xfs_sb_version_haslogxattrs (&mp->m_sb));switch (xfs_attr_intent_op (attr)) { +case XFS_ATTRI_OP_FLAGS_NVREPLACEXXX :case XFS_ATTRI_OP_FLAGS_NVSET : args->op_flags |=XFS_DA_OP_VLOOKUP ; fallthrough;
@@ -745,7 +750,10 @@ xfs_attri_item_relog( new_attrp->alfi_ino = old_attrp->alfi_ino; new_attrp->alfi_op_flags = old_attrp->alfi_op_flags; new_attrp->alfi_value_len = old_attrp->alfi_value_len; -if (xfs_attr_log_item_op (old_attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACE ) { +if (xfs_attr_log_item_op (old_attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACEXXX ) { + new_attrp->alfi_newname_len = old_attrp->alfi_newname_len; + new_attrp->alfi_oldname_len = old_attrp->alfi_oldname_len; + }else if (xfs_attr_log_item_op (old_attrp) ==XFS_ATTRI_OP_FLAGS_NVREPLACE ) { new_attrp->alfi_newname_len = old_attrp->alfi_newname_len; new_attrp->alfi_oldname_len = old_attrp->alfi_oldname_len; }else {
@@ -825,6 +833,16 @@ xlog_recover_attri_commit_pass2( } name_len = attri_formatp->alfi_name_len;break ; +case XFS_ATTRI_OP_FLAGS_NVREPLACEXXX : +/* Log item, attr name, new attr name, optional attr value */ +if (item->ri_total <3 || item->ri_total >4 ) { +XFS_CORRUPTION_ERROR (__func__,XFS_ERRLEVEL_LOW , mp, + attri_formatp, len); +return -EFSCORRUPTED ; + } + name_len = attri_formatp->alfi_oldname_len; + newname_len = attri_formatp->alfi_newname_len; +break ;case XFS_ATTRI_OP_FLAGS_NVREPLACE :/* Log item, attr name, new attr name, attr value */ if (item->ri_total !=4 ) {
@@ -929,6 +947,7 @@ xlog_recover_attri_commit_pass2( }break ;case XFS_ATTRI_OP_FLAGS_NVREPLACE : +case XFS_ATTRI_OP_FLAGS_NVREPLACEXXX :/* * Name-value replace operations require the caller to specify * the old and new name explicitly. Values are optional.
PATCH 16/17 xfs: log new xattr values for NVREPLACEXXX operations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
For NVREPLACEXXX operations, make it possible to log the old and new attr values, since this variant does VLOOKUP operations and we actually need both old and new values to be represented.
--- fs/xfs/libxfs/xfs_attr.c | 4 +- fs/xfs/libxfs/xfs_da_btree.h | 2 + fs/xfs/libxfs/xfs_log_format.h | 14 +++++- fs/xfs/xfs_attr_item.c | 88 +++++++++++++++++++++++++++++++++++----- fs/xfs/xfs_attr_item.h | 3 + 5 files changed, 93 insertions(+), 18 deletions(-)
fs/xfs/libxfs/xfs_attr.c +3 -1
@@ -432,13 +432,15 @@ xfs_attr_complete_op(/* * NVREPLACE operations require the caller to set the old and new names -* explicitly. +* and values explicitly. */ ASSERT (args->new_namelen >0 ); args->name = args->new_name; args->namelen = args->new_namelen; args->hashval =xfs_da_hashname (args->name, args->namelen); + args->value = args->new_value; + args->valuelen = args->new_valuelen;return replace_state; }
fs/xfs/libxfs/xfs_da_btree.h +2 -0
@@ -60,7 +60,9 @@ typedef struct xfs_da_args {int new_namelen;/* new attr name len */ uint8_t filetype;/* filetype of inode for directories */ void *value;/* set of bytes (maybe contain NULLs) */ +void *new_value;/* new xattr value (may contain NULLs) */ int valuelen;/* length of value */ +int new_valuelen;/* length of new value */ unsigned int attr_filter;/* XFS_ATTR_{ROOT,SECURE,INCOMPLETE} */ unsigned int attr_flags;/* XATTR_{CREATE,REPLACE} */ xfs_dahash_t hashval;/* hash value of name */
fs/xfs/libxfs/xfs_log_format.h +10 -4
@@ -115,11 +115,11 @@ struct xfs_unmount_log_format {#define XLOG_REG_TYPE_BUD_FORMAT 26#define XLOG_REG_TYPE_ATTRI_FORMAT 27#define XLOG_REG_TYPE_ATTRD_FORMAT 28 -#define XLOG_REG_TYPE_ATTR_NAME 29 +#define XLOG_REG_TYPE_ATTR_NAME 29#define XLOG_REG_TYPE_ATTR_VALUE 30#define XLOG_REG_TYPE_ATTR_NEWNAME 31 -#define XLOG_REG_TYPE_MAX 31 - +#define XLOG_REG_TYPE_ATTR_NEWVALUE 32 +#define XLOG_REG_TYPE_MAX 32/* * Flags to log operation header
@@ -980,7 +980,13 @@ struct xfs_icreate_log {struct xfs_attri_log_format {uint16_t alfi_type;/* attri log item type */ uint16_t alfi_size;/* size of this item */ -uint32_t __pad;/* pad to 64 bit aligned */ + +/* +* For NVREPLACEXXX, this is the length of the new xattr value. +* alfi_value_len contains the length of the old xattr value. +*/ +uint32_t alfi_newvalue_len; +uint64_t alfi_id;/* attri identifier */ uint64_t alfi_ino;/* the inode for this attr operation */ uint32_t alfi_op_flags;/* marks the op as a set or remove */
fs/xfs/xfs_attr_item.c +76 -12
239 lines not on this page · raw message
fs/xfs/xfs_attr_item.h +2 -1
@@ -13,8 +13,9 @@ struct kmem_zone;struct xfs_attri_log_nameval {struct xfs_log_iovec name; -struct xfs_log_iovec newname; +struct xfs_log_iovec newname;/* NVREPLACEXXX only */ struct xfs_log_iovec value; +struct xfs_log_iovec newvalue;/* NVREPLACEXXX only */ refcount_t refcount;/* name and value follow the end of this struct */
PATCH 17/17 xfs: replace parent pointer diroffset with full dirent name
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
As I've mentioned in past comments on the parent pointers patchset, the proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:
(parent_ino, parent_gen, diroffset) -> (dirent_name)
If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. This is difficult to guarantee in the process of performing a repair and in the long terms is totally pointless because the the parent pointer code doesn't use the diroffset, nor does the online fsck code. IOWs, this design decision increases the complexity of repair solely to work around the xattr code being unable to store a full dirent name in the xattr name or match on xattr values.
However, xattrs can now perform lookups on both the name and value of an xattr. This means we can redefine the parent pointer format like this:
For dirent names shorter than 243 bytes:
(parent_ino, parent_gen, dirent_name) -> NULL
For dirent names longer than 243 bytes:
(parent_ino, parent_gen, dirent_name[0:242]) -> (dirent_name[243:255])
and this works because a parent pointer lookup uses NVLOOKUP, which will read those last twelve bytes of the dirent name if need be.
Replace the diroffset with this new format, thereby eliminating the need for directory repair to update all the parent pointers after rebuilding the directory. We haven't merged parent pointers, so changes to the ondisk format are easy.
--- fs/xfs/libxfs/xfs_da_format.h | 39 ++++++- fs/xfs/libxfs/xfs_fs.h | 2 fs/xfs/libxfs/xfs_parent.c | 216 +++++++++++++++++++++++++--------------- fs/xfs/libxfs/xfs_parent.h | 46 +++++---- fs/xfs/libxfs/xfs_trans_resv.c | 7 + fs/xfs/scrub/dir.c | 34 +----- fs/xfs/scrub/dir_repair.c | 87 +++++----------- fs/xfs/scrub/parent.c | 48 ++------- fs/xfs/scrub/parent_repair.c | 55 +++------- fs/xfs/scrub/trace.h | 65 ++---------- fs/xfs/xfs_inode.c | 30 ++---- fs/xfs/xfs_ondisk.h | 1 fs/xfs/xfs_parent_utils.c | 8 + fs/xfs/xfs_symlink.c | 3 - 14 files changed, 291 insertions(+), 350 deletions(-)
fs/xfs/libxfs/xfs_da_format.h +34 -5
@@ -827,14 +827,43 @@ xfs_failaddr_t xfs_da3_blkinfo_verify(struct xfs_buf *bp,/* * Parent pointer attribute format definition * -* EA name encodes the parent inode number, generation and the offset of -* the dirent that points to the child inode. The EA value contains the -* same name as the dirent in the parent directory. +* The EA name encodes the parent inode number, generation and as much of the +* dirent name as fits. In other words, it contains up to 243 bytes of the +* dirent name. +* +* The EA value contains however much of the dirent name that does not fit in +* the EA name. */ struct xfs_parent_name_rec {__be64 p_ino;__be32 p_gen; -__be32 p_diroffset; -}; +__u8 p_dname[]; +} __attribute__((packed)); + +/* Maximum size of a parent pointer EA name. */ +#define XFS_PARENT_NAME_MAX_SIZE \ + (MAXNAMELEN - 1) + +/* Maximum number of dirent name bytes stored in p_dname. */ +#define XFS_PARENT_MAX_DNAME_SIZE \ + (XFS_PARENT_NAME_MAX_SIZE - sizeof(struct xfs_parent_name_rec)) + +/* Maximum number of dirent name bytes stored in the xattr value. */ +#define XFS_PARENT_MAX_DNAME_VALUELEN \ + sizeof(struct xfs_parent_name_rec) + +static inline unsigned int +xfs_parent_name_rec_sizeof ( +unsigned int dnamelen) +{ +return sizeof (struct xfs_parent_name_rec ) + dnamelen; +} + +static inline unsigned int +xfs_parent_name_dnamelen ( +unsigned int rec_sizeof) +{ +return rec_sizeof -sizeof (struct xfs_parent_name_rec ); +}#endif /* __XFS_DA_FORMAT_H__ */
fs/xfs/libxfs/xfs_fs.h +1 -1
@@ -769,7 +769,7 @@ struct xfs_scrub_metadata {struct xfs_getparents_rec {__u64 gpr_ino;/* Inode number */ __u32 gpr_gen;/* Inode generation */ -__u32 gpr_diroffset;/* Directory offset */ +__u32 gpr_rsvd2;/* Reserved */ __u64 gpr_rsvd;/* Reserved */ __u8 gpr_name[];/* File name and null terminator */ };
fs/xfs/libxfs/xfs_parent.c +136 -80
385 lines not on this page · raw message
fs/xfs/libxfs/xfs_parent.h +26 -20
88 lines not on this page · raw message
fs/xfs/libxfs/xfs_trans_resv.c +4 -3
@@ -427,19 +427,20 @@ static inline unsigned int xfs_calc_pptr_link_overhead(void) {return sizeof (struct xfs_attri_log_format ) +xlog_calc_iovec_len (XATTR_NAME_MAX ) + -xlog_calc_iovec_len (sizeof (struct xfs_parent_name_rec )); +xlog_calc_iovec_len (XFS_PARENT_NAME_MAX_SIZE ); }static inline unsigned int xfs_calc_pptr_unlink_overhead (void ) {return sizeof (struct xfs_attri_log_format ) + -xlog_calc_iovec_len (sizeof (struct xfs_parent_name_rec )); +xlog_calc_iovec_len (XFS_PARENT_NAME_MAX_SIZE ); }static inline unsigned int xfs_calc_pptr_replace_overhead (void ) {return sizeof (struct xfs_attri_log_format ) +xlog_calc_iovec_len (XATTR_NAME_MAX ) +xlog_calc_iovec_len (XATTR_NAME_MAX ) + -xlog_calc_iovec_len (sizeof (struct xfs_parent_name_rec )); +xlog_calc_iovec_len (XFS_PARENT_NAME_MAX_SIZE ) + +xlog_calc_iovec_len (XFS_PARENT_NAME_MAX_SIZE ); }/*
fs/xfs/scrub/dir.c +8 -26
@@ -78,7 +78,7 @@ struct xchk_dir {/* If we've cycled the ILOCK, we must revalidate deferred dirents. */ bool need_revalidate; -/* Name buffer for pptr validation and dirent revalidation. */ +/* Name buffer for dirent revalidation. */ uint8_t namebuf[MAXNAMELEN ]; };
@@ -139,38 +139,20 @@ xchk_dir_lock_child( STATIC intxchk_dir_parent_pointer (struct xchk_dir *sd, -xfs_dir2_dataptr_t dapos,const struct xfs_name *name,struct xfs_inode *ip) {struct xfs_scrub *sc = sd->sc; -int pptr_namelen; +int error; sd->pptr.p_ino = sc->ip->i_ino; sd->pptr.p_gen =VFS_I (sc->ip)->i_generation; - sd->pptr.p_diroffset = dapos; + sd->pptr.p_namelen = name->len; +memcpy (sd->pptr.p_name, name->name, name->len); - pptr_namelen =xfs_parent_lookup (sc->tp, ip, &sd->pptr, sd->namebuf, -MAXNAMELEN , &sd->pptr_scratch); -if (pptr_namelen == -ENOATTR ) { + error =xfs_parent_lookup (sc->tp, ip, &sd->pptr, &sd->pptr_scratch); +if (error == -ENOATTR )xchk_fblock_xref_set_corrupt (sc,XFS_DATA_FORK ,0 ); -return 0 ; - } -if (pptr_namelen <0 ) { -xchk_fblock_xref_process_error (sc,XFS_DATA_FORK ,0 , - &pptr_namelen); -return pptr_namelen; - } - -if (pptr_namelen != name->len) { -xchk_fblock_xref_set_corrupt (sc,XFS_DATA_FORK ,0 ); -return 0 ; - } - -if (memcmp (sd->namebuf, name->name, name->len)) { -xchk_fblock_xref_set_corrupt (sc,XFS_DATA_FORK ,0 ); -return 0 ; - }return 0 ; }
@@ -216,7 +198,7 @@ xchk_dir_check_pptr_fast(return 0 ; } - error =xchk_dir_parent_pointer (sd, dapos, name, ip); + error =xchk_dir_parent_pointer (sd, name, ip);xfs_iunlock (ip, lockmode);return error; }
@@ -1041,7 +1023,7 @@ xchk_dir_slow_dirent( goto out_unlock; check_pptr: - error =xchk_dir_parent_pointer (sd, dirent->diroffset, &xname, ip); + error =xchk_dir_parent_pointer (sd, &xname, ip); out_unlock:xfs_iunlock (ip, lockmode); out_rele:
fs/xfs/scrub/dir_repair.c +27 -60
255 lines not on this page · raw message
fs/xfs/scrub/parent.c +10 -38
116 lines not on this page · raw message
fs/xfs/scrub/parent_repair.c +15 -40
140 lines not on this page · raw message
fs/xfs/scrub/trace.h +11 -54
153 lines not on this page · raw message
fs/xfs/xfs_inode.c +12 -18
@@ -1202,8 +1202,7 @@ xfs_create(* the parent information now. */ if (parent) { - error =xfs_parent_add (tp, parent, dp, name, diroffset, - ip); + error =xfs_parent_add (tp, parent, dp, name, ip);if (error) goto out_trans_cancel; }
@@ -1477,8 +1476,7 @@ xfs_link(* the parent to the inode. */ if (parent) { - error =xfs_parent_add (tp, parent, tdp, target_name, - diroffset, sip); + error =xfs_parent_add (tp, parent, tdp, target_name, sip);if (error) goto error_return; }
@@ -2750,7 +2748,7 @@ xfs_remove( }if (parent) { - error =xfs_parent_remove (tp, dp, parent, dir_offset, ip); + error =xfs_parent_remove (tp, parent, dp, name, ip);if (error) goto out_trans_cancel; }
@@ -3061,13 +3059,13 @@ xfs_cross_rename( }if (xfs_has_parent (mp)) { - error =xfs_parent_replace (tp, ip1_pptr, dp1, - old_diroffset, name2, dp2, new_diroffset, ip1); + error =xfs_parent_replace (tp, ip1_pptr, dp1, name1, dp2, + name2, ip1);if (error) goto out_trans_abort; - error =xfs_parent_replace (tp, ip2_pptr, dp2, - new_diroffset, name1, dp1, old_diroffset, ip2); + error =xfs_parent_replace (tp, ip2_pptr, dp2, name2, dp1, + name1, ip2);if (error) goto out_trans_abort; }
@@ -3540,25 +3538,21 @@ xfs_rename( goto out_trans_cancel;if (wip_pptr) { - error =xfs_parent_add (tp, wip_pptr, - src_dp, src_name, - old_diroffset, wip); + error =xfs_parent_add (tp, wip_pptr, src_dp, src_name, wip);if (error) goto out_trans_cancel; }if (src_ip_pptr) { - error =xfs_parent_replace (tp, src_ip_pptr, src_dp, - old_diroffset, target_name, target_dp, - new_diroffset, src_ip); + error =xfs_parent_replace (tp, src_ip_pptr, src_dp, src_name, + target_dp, target_name, src_ip);if (error) goto out_trans_cancel; }if (tgt_ip_pptr) { - error =xfs_parent_remove (tp, target_dp, - tgt_ip_pptr, - new_diroffset, target_ip); + error =xfs_parent_remove (tp, tgt_ip_pptr, target_dp, + target_name, target_ip);if (error) goto out_trans_cancel; }
fs/xfs/xfs_ondisk.h +1 -0
@@ -114,6 +114,7 @@ xfs_check_ondisk_structs(void)XFS_CHECK_OFFSET (xfs_dir2_sf_entry_t, offset,1 );XFS_CHECK_OFFSET (xfs_dir2_sf_entry_t, name,3 );XFS_CHECK_STRUCT_SIZE (xfs_dir2_sf_hdr_t,10 ); + XFS_CHECK_STRUCT_SIZE(struct xfs_parent_name_rec, 12);/* log structures */ XFS_CHECK_STRUCT_SIZE(struct xfs_buf_log_format, 88);
fs/xfs/xfs_parent_utils.c +5 -3
@@ -67,12 +67,14 @@ xfs_getparent_listent(* pointer. The attr list functions filtered out INCOMPLETE attrs. */ if (XFS_IS_CORRUPT (mp, !xfs_parent_namecheck (mp, rec, namelen, flags)) || -XFS_IS_CORRUPT (mp, !xfs_parent_valuecheck (mp, value, valuelen))) { +XFS_IS_CORRUPT (mp, !xfs_parent_valuecheck (mp, namelen, value, + valuelen))) { context->seen_enough = -EFSCORRUPTED ;return ; } -xfs_parent_irec_from_disk (&gp->pptr_irec, rec, value, valuelen); +xfs_parent_irec_from_disk (&gp->pptr_irec, rec, namelen, value, + valuelen);/* * We found a parent pointer, but we've filled up the buffer. Signal
@@ -93,7 +95,7 @@ xfs_getparent_listent( pptr =xfs_getparents_rec (ppi, ppi->gp_count); pptr->gpr_ino = irec->p_ino; pptr->gpr_gen = irec->p_gen; - pptr->gpr_diroffset = irec->p_diroffset; + pptr->gpr_rsvd2 =0 ; pptr->gpr_rsvd =0 ;memcpy (pptr->gpr_name, irec->p_name, irec->p_namelen);
fs/xfs/xfs_symlink.c +1 -2
@@ -348,8 +348,7 @@ xfs_symlink(xfs_trans_log_inode (tp, dp,XFS_ILOG_CORE );if (parent) { - error =xfs_parent_add (tp, parent, dp, link_name, - diroffset, ip); + error =xfs_parent_add (tp, parent, dp, link_name, ip);if (error) goto out_trans_cancel; }
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
In reply to Darrick J. WongQuoted 84 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(
parent_ino,parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(
parent_ino,dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(
parent_ino,parent_gen,dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.The ondisk format of a parent pointer can then become:
(
parent_ino,parent_gen, dirent_name[0:242]) -> (dirent_name[243:255])Because we can always look up a specific parent pointer. Most of the patches in this patchset prepare the high level xattr code and the lower
level logging code to do this correctly, and the last patch switches the
ondisk format.
Quoted 6 lines
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. kernel git tree:
So I gave this set a look over and for the most part I think it's ok as long as folks agree on the new format? It's a lot different from what folks originally articulated that they wanted, but for the most part the format change doesnt affect parent pointer mechanics as much as it will affect other features that may use it. I havnt seen much complaints in response, so if it's all the same to everyone else, I am fine with moving forward with it? Thanks all!
Allison
Quoted 36 lines
fstests git tree:
https://urldefense.com/v3/__https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key__;!!ACWV5N9M2RV99hQ!NDopQXFVRjDfqgJvGMQRzS3fXyyd1SC4HfGl4dxSMWxNPsNKScecjp7bZYX-dDhL5tatk_xxJFdQwI5iKKgk$
---
fs/xfs/libxfs/xfs_attr.c | 66 +++++--- fs/xfs/libxfs/xfs_attr.h | 5 + fs/xfs/libxfs/xfs_attr_leaf.c | 41 ++++- fs/xfs/libxfs/xfs_da_btree.h | 6 + fs/xfs/libxfs/xfs_da_format.h | 39 ++++- fs/xfs/libxfs/xfs_fs.h | 2fs/xfs/libxfs/xfs_log_format.h| 31 +++- fs/xfs/libxfs/xfs_parent.c | 215 +++++++++++++++++---------- fs/xfs/libxfs/xfs_parent.h | 46 +++---fs/xfs/libxfs/xfs_trans_resv.c| 7 + fs/xfs/scrub/dir.c | 34 +--- fs/xfs/scrub/dir_repair.c | 87 +++-------- fs/xfs/scrub/parent.c | 48 +----- fs/xfs/scrub/parent_repair.c | 55 ++----- fs/xfs/scrub/trace.h | 65 +------- fs/xfs/xfs_attr_item.c | 318
+++++++++++++++++++++++++++++++---------
fs/xfs/xfs_attr_item.h | 3 fs/xfs/xfs_inode.c | 30 ++-- fs/xfs/xfs_ondisk.h | 1 fs/xfs/xfs_parent_utils.c | 8 + fs/xfs/xfs_symlink.c | 3 fs/xfs/xfs_xattr.c | 5 +22 files changed, 660 insertions(+), 455 deletions(-)
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 74 lines · Allison Henderson
Quoted 72 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(parent_ino, parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(parent_ino, parent_gen, dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.
Quoted 4 lines
Quoted 4 lines
The ondisk format of a parent pointer can then become:
(parent_ino, parent_gen, dirent_name[0:242]) ->(dirent_name[243:255])
With VLOOKUP in place, why is this better than
(parent_ino, parent_gen) -> (dirent_name)
Is it because the dabtree hash is calculated only on the key and changing that would be way more intrusive?
Does that mean that user can create up to 2^12 parent pointers with the same hash and we are fine with it?
I don't think it is a problem, just wanted to understand and ask that the reason for this part in the format be explained.
Thanks,
Amir.
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 78 lines · Amir Goldstein
Quoted 76 lines · Allison Henderson
Quoted 74 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(parent_ino, parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(parent_ino, parent_gen, dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.The ondisk format of a parent pointer can then become:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
(parent_ino, parent_gen, dirent_name[0:242]) ->(dirent_name[243:255])
With VLOOKUP in place, why is this better than
(parent_ino,parent_gen) -> (dirent_name)Is it because the dabtree hash is calculated only on the key and changing that would be way more intrusive?
Yes. The dabtree hash is calculated on the full attr name, so this is an attempt to reduce hash collisions during parent pointer lookups by stuffing as many bytes in the attr name as possible.
It would be very easy to change the xfs_da_hashname calls inside xfs_parent.c to use either the full dirent name (i.e. pptr hash matches dirent hash) or use the full parent pointer and not just the attr key, but that would be a major break from the tradition that the da hash is computed against the attr name only.
(I'm not opposed to doing that too, but one breaking change at a time. ;))
Quoted 2 lines
Does that mean that user can create up to 2^12 parent pointers with the same hash and we are fine with it?
Yes. The dabtree can handle an xattr structure where every name hashes to the same value, though performance will be slow as it scans every attr to find the one it wants.
The number of parent pointers with the same hash can be much higher than 2^12 -- I wrote a dumb C program that creates an arbitrary number of attr names with identical hashes. It's not fast when you get past 100,000. :P
Quoted 2 lines
I don't think it is a problem, just wanted to understand and ask that the reason for this part in the format be explained.
NP. Thanks for asking!
--D
Quoted 2 lines
Thanks,
Amir.
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 105 lines · Darrick J. Wong
Quoted 78 lines · Amir Goldstein
Quoted 76 lines · Allison Henderson
Quoted 74 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(parent_ino, parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(parent_ino, parent_gen, dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.The ondisk format of a parent pointer can then become:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
(parent_ino, parent_gen, dirent_name[0:242]) ->(dirent_name[243:255])
With VLOOKUP in place, why is this better than
(parent_ino,parent_gen) -> (dirent_name)Is it because the dabtree hash is calculated only on the key and changing that would be way more intrusive?
Yes. The dabtree hash is calculated on the full attr name, so this is an attempt to reduce hash collisions during parent pointer lookups by stuffing as many bytes in the attr name as possible.
It would be very easy to change the
xfs_da_hashnamecalls insidexfs_parent.cto use either the full dirent name (i.e. pptr hash matches dirent hash) or use the full parent pointer and not just the attr key, but that would be a major break from the tradition that the da hash is computed against the attr name only.(I'm not opposed to doing that too, but one breaking change at a time. ;))
Quoted 2 lines
Does that mean that user can create up to 2^12 parent pointers with the same hash and we are fine with it?
Quoted 9 lines
Yes. The dabtree can handle an xattr structure where every name hashes to the same value, though performance will be slow as it scans every attr to find the one it wants.
The number of parent pointers with the same hash can be much higher than 2^12 -- I wrote a dumb C program that creates an arbitrary number of attr names with identical hashes. It's not fast when you get past 100,000. :P
Right.
So how about
(parent_ino, parent_gen, dirent_hash) -> (dirent_name)
This is not a breaking change and you won't need to do another breaking change later.
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
Thanks,
Amir.
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 120 lines · Amir Goldstein
Quoted 105 lines · Darrick J. Wong
Quoted 78 lines · Amir Goldstein
Quoted 76 lines · Allison Henderson
Quoted 74 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(parent_ino, parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(parent_ino, parent_gen, dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.The ondisk format of a parent pointer can then become:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
(parent_ino, parent_gen, dirent_name[0:242]) ->(dirent_name[243:255])
With VLOOKUP in place, why is this better than
(parent_ino,parent_gen) -> (dirent_name)Is it because the dabtree hash is calculated only on the key and changing that would be way more intrusive?
Yes. The dabtree hash is calculated on the full attr name, so this is an attempt to reduce hash collisions during parent pointer lookups by stuffing as many bytes in the attr name as possible.
It would be very easy to change the
xfs_da_hashnamecalls insidexfs_parent.cto use either the full dirent name (i.e. pptr hash matches dirent hash) or use the full parent pointer and not just the attr key, but that would be a major break from the tradition that the da hash is computed against the attr name only.(I'm not opposed to doing that too, but one breaking change at a time. ;))
Quoted 2 lines
Does that mean that user can create up to 2^12 parent pointers with the same hash and we are fine with it?
Quoted 9 lines
Yes. The dabtree can handle an xattr structure where every name hashes to the same value, though performance will be slow as it scans every attr to find the one it wants.
The number of parent pointers with the same hash can be much higher than 2^12 -- I wrote a dumb C program that creates an arbitrary number of attr names with identical hashes. It's not fast when you get past 100,000. :P
Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)
Quoted 5 lines
This is not a breaking change and you won't need to do another breaking change later.
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
I wrote an xfs_io command to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
Hopefully the fact that the attr name starts with 12 bytes (4 of which aren't known to unprivileged userspace) and omits the last 12 bytes of the dirent name will make it harder to generate double-collisions. Granted there's probably someone who's more math-smart than me who can figure this out.
--D
Quoted 2 lines
Thanks,
Amir.
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 140 lines · Darrick J. Wong
Quoted 120 lines · Amir Goldstein
Quoted 105 lines · Darrick J. Wong
Quoted 78 lines · Amir Goldstein
Quoted 76 lines · Allison Henderson
Quoted 74 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(parent_ino, parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(parent_ino, parent_gen, dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.The ondisk format of a parent pointer can then become:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
(parent_ino, parent_gen, dirent_name[0:242]) ->(dirent_name[243:255])
With VLOOKUP in place, why is this better than
(parent_ino,parent_gen) -> (dirent_name)Is it because the dabtree hash is calculated only on the key and changing that would be way more intrusive?
Yes. The dabtree hash is calculated on the full attr name, so this is an attempt to reduce hash collisions during parent pointer lookups by stuffing as many bytes in the attr name as possible.
It would be very easy to change the
xfs_da_hashnamecalls insidexfs_parent.cto use either the full dirent name (i.e. pptr hash matches dirent hash) or use the full parent pointer and not just the attr key, but that would be a major break from the tradition that the da hash is computed against the attr name only.(I'm not opposed to doing that too, but one breaking change at a time. ;))
Quoted 2 lines
Does that mean that user can create up to 2^12 parent pointers with the same hash and we are fine with it?
Quoted 9 lines
Yes. The dabtree can handle an xattr structure where every name hashes to the same value, though performance will be slow as it scans every attr to find the one it wants.
The number of parent pointers with the same hash can be much higher than 2^12 -- I wrote a dumb C program that creates an arbitrary number of attr names with identical hashes. It's not fast when you get past 100,000. :P
Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)Quoted 5 lines
This is not a breaking change and you won't need to do another breaking change later.
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
I wrote an
xfs_iocommand to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.
Quoted 7 lines
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
OK, but this attack is applicable to the dabtree hash itself only with the proposed dirent_name[0:242] key, is it not?
Quoted 5 lines
Hopefully the fact that the attr name starts with 12 bytes (4 of which aren't known to unprivileged userspace) and omits the last 12 bytes of the dirent name will make it harder to generate double-collisions. Granted there's probably someone who's more math-smart than me who can figure this out.
My concern is that generating 7^12 collisions using all the possible printable suffix dirent_name[243:255] is so simple that even a non-malicious but rather unlucky script would end up hitting those collisions:
Report_$ORGANIZATION_$DEPARTEMT_$PRODUCT_$PROJECT_... $GROUP_$OWNER_$DATE_$TIME_$ID.pdf
You get what I mean...
So my thinking was that the unlucky should not suffer the same penalty as the malicious.
Thanks,
Amir.
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
In reply to Darrick J. WongQuoted 12 lines · Darrick J. Wong
Quoted 11 lines · Amir Goldstein
Quoted 3 lines · Darrick J. Wong
Quoted 2 lines · Amir Goldstein
On Fri, Mar 24, 2023 at 8:19 PM Allison Henderson
<[email protected]> wrote:Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)This is not a breaking change and you won't need to do another breaking change later.
Quoted 7 lines
Quoted 2 lines
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
Right, that's because hash collision resolution is very dumb. It's just a linear walk from start to end comparing the names of each coliding entry.
Quoted 6 lines
I wrote an
xfs_iocommand to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.
That's worst case behaviour, not real-world production behaviour. It's also a dabtree optimisation problem, not a parent pointer problem. i.e. if this is a real world issue, we need to fix the dabtree indexing, not work around it in the parent pointer format.
IOWs, optimising the parent pointer structure for the *rare* corner case of an intentionally crafted dahash btree collision farm is, to me, the wrong way to design a structure.
Optimising it for the common fast path (single parent xattrs, followed by small numbers of parents) is what we should be doing, because that situation will occur an uncountable number of times more frequently than this theorectical worst case behaviour. i.e. the worst case will, in all likelihood, never happen in production environments - behaviour in this circumstance is only a consideration for mitigating malicious attacks.
And, quite frankly, nobody is going to bother attacking a filesystem this way - there is literally nothing that can be gained from slowing down access to a single directory in this manner. It's a DOS at best, but there are *much* easier ways of doing that and it doesn't require parent pointers or dahash collisions at all.
Quoted 7 lines
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
So why not change the dahash for parent pointer enabled filesystems to use crc32c everywhere?
I'd much prefer we have a parent pointer index key that is fixed length, has constant hash time and a dabtree hash that is far more resilient to collision farming than to have to perform encoding/decoding of varaible length values into the key to work around a collision-rich hashing algorithm.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 64 lines · Dave Chinner
Quoted 12 lines · Darrick J. Wong
Quoted 11 lines · Amir Goldstein
Quoted 3 lines · Darrick J. Wong
Quoted 2 lines · Amir Goldstein
On Fri, Mar 24, 2023 at 8:19 PM Allison Henderson
<[email protected]> wrote:Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)This is not a breaking change and you won't need to do another breaking change later.
Quoted 7 lines
Quoted 2 lines
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
Right, that's because hash collision resolution is very dumb. It's just a linear walk from start to end comparing the names of each coliding entry.
Quoted 6 lines
I wrote an
xfs_iocommand to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.That's worst case behaviour, not real-world production behaviour. It's also a dabtree optimisation problem, not a parent pointer problem. i.e. if this is a real world issue, we need to fix the dabtree indexing, not work around it in the parent pointer format.
IOWs, optimising the parent pointer structure for the *rare* corner case of an intentionally crafted dahash btree collision farm is, to me, the wrong way to design a structure.
Optimising it for the common fast path (single parent xattrs, followed by small numbers of parents) is what we should be doing, because that situation will occur an uncountable number of times more frequently than this theorectical worst case behaviour. i.e. the worst case will, in all likelihood, never happen in production environments - behaviour in this circumstance is only a consideration for mitigating malicious attacks.
And, quite frankly, nobody is going to bother attacking a filesystem this way - there is literally nothing that can be gained from slowing down access to a single directory in this manner. It's a DOS at best, but there are *much* easier ways of doing that and it doesn't require parent pointers or dahash collisions at all.
Quoted 7 lines
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
Quoted 2 lines
So why not change the dahash for parent pointer enabled filesystems to use crc32c everywhere?
That's not difficult to do, but it'll break name obscuring in metadump unless you know of a quick way to derive B from A and maintain crc32c(A) == crc32c(B).
Granted I had mostly written off name obscuring in metadump on parent pointer filesystems anyway.
Quoted 5 lines
I'd much prefer we have a parent pointer index key that is fixed length, has constant hash time and a dabtree hash that is far more resilient to collision farming than to have to perform encoding/decoding of varaible length values into the key to work around a collision-rich hashing algorithm.
Wellllll I was 10 minutes away from sending v11 with all of my changes integrated, but I guess now I'll go rework the dabtree to use crc32c, encode the crc32c in the parent pointer attr name, and get rid of all the variable length value decoding. Now that I just got rid of all the intermediate patches with hash-in-attr-name.
Let's see if I can hammer that out in the next 15 minutes.
--D
Quoted 6 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 65 lines · Darrick J. Wong
Quoted 64 lines · Dave Chinner
Quoted 12 lines · Darrick J. Wong
Quoted 11 lines · Amir Goldstein
Quoted 3 lines · Darrick J. Wong
Quoted 2 lines · Amir Goldstein
On Fri, Mar 24, 2023 at 8:19 PM Allison Henderson
<[email protected]> wrote:Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)This is not a breaking change and you won't need to do another breaking change later.
Quoted 7 lines
Quoted 2 lines
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
Right, that's because hash collision resolution is very dumb. It's just a linear walk from start to end comparing the names of each coliding entry.
Quoted 6 lines
I wrote an
xfs_iocommand to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.That's worst case behaviour, not real-world production behaviour. It's also a dabtree optimisation problem, not a parent pointer problem. i.e. if this is a real world issue, we need to fix the dabtree indexing, not work around it in the parent pointer format.
IOWs, optimising the parent pointer structure for the *rare* corner case of an intentionally crafted dahash btree collision farm is, to me, the wrong way to design a structure.
Optimising it for the common fast path (single parent xattrs, followed by small numbers of parents) is what we should be doing, because that situation will occur an uncountable number of times more frequently than this theorectical worst case behaviour. i.e. the worst case will, in all likelihood, never happen in production environments - behaviour in this circumstance is only a consideration for mitigating malicious attacks.
And, quite frankly, nobody is going to bother attacking a filesystem this way - there is literally nothing that can be gained from slowing down access to a single directory in this manner. It's a DOS at best, but there are *much* easier ways of doing that and it doesn't require parent pointers or dahash collisions at all.
Quoted 7 lines
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
Quoted 6 lines
Quoted 2 lines
So why not change the dahash for parent pointer enabled filesystems to use crc32c everywhere?
That's not difficult to do, but it'll break name obscuring in metadump unless you know of a quick way to derive B from A and maintain crc32c(A) == crc32c(B).
Not easily, but....
Quoted 2 lines
Granted I had mostly written off name obscuring in metadump on parent pointer filesystems anyway.
... so had I. The name obfuscation needs a completely different approach for PP enabled filesystems because of the name also being embedded in non-user visible structures and potentially changing attr tree dahash-based indexing.
Quoted 11 lines
Quoted 5 lines
I'd much prefer we have a parent pointer index key that is fixed length, has constant hash time and a dabtree hash that is far more resilient to collision farming than to have to perform encoding/decoding of varaible length values into the key to work around a collision-rich hashing algorithm.
Wellllll I was 10 minutes away from sending v11 with all of my changes integrated, but I guess now I'll go rework the dabtree to use crc32c, encode the crc32c in the parent pointer attr name, and get rid of all the variable length value decoding. Now that I just got rid of all the intermediate patches with hash-in-attr-name.
I apologise for not being able to comment on this earlier. What time I've had available over the past coule of weeks has been focussed on the parts of the repair patchset that are ready for merge and I've had little time to follow the progress of the parts you are currently developing.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 75 lines · Dave Chinner
Quoted 65 lines · Darrick J. Wong
Quoted 64 lines · Dave Chinner
Quoted 12 lines · Darrick J. Wong
Quoted 11 lines · Amir Goldstein
Quoted 3 lines · Darrick J. Wong
Quoted 2 lines · Amir Goldstein
On Fri, Mar 24, 2023 at 8:19 PM Allison Henderson
<[email protected]> wrote:Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)This is not a breaking change and you won't need to do another breaking change later.
Quoted 7 lines
Quoted 2 lines
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
Right, that's because hash collision resolution is very dumb. It's just a linear walk from start to end comparing the names of each coliding entry.
Quoted 6 lines
I wrote an
xfs_iocommand to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.That's worst case behaviour, not real-world production behaviour. It's also a dabtree optimisation problem, not a parent pointer problem. i.e. if this is a real world issue, we need to fix the dabtree indexing, not work around it in the parent pointer format.
IOWs, optimising the parent pointer structure for the *rare* corner case of an intentionally crafted dahash btree collision farm is, to me, the wrong way to design a structure.
Optimising it for the common fast path (single parent xattrs, followed by small numbers of parents) is what we should be doing, because that situation will occur an uncountable number of times more frequently than this theorectical worst case behaviour. i.e. the worst case will, in all likelihood, never happen in production environments - behaviour in this circumstance is only a consideration for mitigating malicious attacks.
And, quite frankly, nobody is going to bother attacking a filesystem this way - there is literally nothing that can be gained from slowing down access to a single directory in this manner. It's a DOS at best, but there are *much* easier ways of doing that and it doesn't require parent pointers or dahash collisions at all.
Quoted 7 lines
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
Quoted 6 lines
Quoted 2 lines
So why not change the dahash for parent pointer enabled filesystems to use crc32c everywhere?
That's not difficult to do, but it'll break name obscuring in metadump unless you know of a quick way to derive B from A and maintain crc32c(A) == crc32c(B).
Not easily, but....
Quoted 7 lines
Quoted 2 lines
Granted I had mostly written off name obscuring in metadump on parent pointer filesystems anyway.
... so had I. The name obfuscation needs a completely different approach for PP enabled filesystems because of the name also being embedded in non-user visible structures and potentially changing attr tree dahash-based indexing.
<nod> I've wondered if we just have to record (dir_ino/name) -> (newname) and use that to update the parent pointer and any dabtree entries we find. That's going to be a bit of a mess to iron out since we can't change the "mounted" filesystem.
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
I'd much prefer we have a parent pointer index key that is fixed length, has constant hash time and a dabtree hash that is far more resilient to collision farming than to have to perform encoding/decoding of varaible length values into the key to work around a collision-rich hashing algorithm.
Wellllll I was 10 minutes away from sending v11 with all of my changes integrated, but I guess now I'll go rework the dabtree to use crc32c, encode the crc32c in the parent pointer attr name, and get rid of all the variable length value decoding. Now that I just got rid of all the intermediate patches with hash-in-attr-name.
Quoted 5 lines
I apologise for not being able to comment on this earlier. What time I've had available over the past coule of weeks has been focussed on the parts of the repair patchset that are ready for merge and I've had little time to follow the progress of the parts you are currently developing.
<nod> Sorry, that was my kneejerk reaction to "let's go reopen this thing and go back to where you were a week ago". I appreciate yuor help getting the online repair patchset ready for merge.
In the end it took about half an hour to put everything back, so I'll go test that overnight and let's see where we end up tomorrow.
--D
Quoted 4 lines
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
Quoted 67 lines · Darrick J. Wong
Quoted 66 lines · Dave Chinner
Quoted 65 lines · Darrick J. Wong
Quoted 64 lines · Dave Chinner
Quoted 12 lines · Darrick J. Wong
Quoted 11 lines · Amir Goldstein
Quoted 3 lines · Darrick J. Wong
Quoted 2 lines · Amir Goldstein
On Fri, Mar 24, 2023 at 8:19 PM Allison Henderson
<[email protected]> wrote:Right.
So how about
(parent_ino,parent_gen,dirent_hash) -> (dirent_name)This is not a breaking change and you won't need to do another breaking change later.
Quoted 7 lines
Quoted 2 lines
This could also be internal to VLOOKUP: appended vhash to attr name and limit VLOOKUP name size to 255 - vhashsize.
The original "put the hash in the xattr name" patches did that, but I discarded that approach because it increases the size of each parent pointer by 4 bytes, and was really easy to make a verrrry slow filesystem:
Right, that's because hash collision resolution is very dumb. It's just a linear walk from start to end comparing the names of each coliding entry.
Quoted 6 lines
I wrote an
xfs_iocommand to compute lists of names with the same dirhash value. If I created a giant directory with the same file hardlinked millions of times where all those dirent names all hash to the same value, lookups in the directory gets really slow because the dabtree code has to walk (on average) half a million dirents to find the matching one.That's worst case behaviour, not real-world production behaviour. It's also a dabtree optimisation problem, not a parent pointer problem. i.e. if this is a real world issue, we need to fix the dabtree indexing, not work around it in the parent pointer format.
IOWs, optimising the parent pointer structure for the *rare* corner case of an intentionally crafted dahash btree collision farm is, to me, the wrong way to design a structure.
Optimising it for the common fast path (single parent xattrs, followed by small numbers of parents) is what we should be doing, because that situation will occur an uncountable number of times more frequently than this theorectical worst case behaviour. i.e. the worst case will, in all likelihood, never happen in production environments - behaviour in this circumstance is only a consideration for mitigating malicious attacks.
And, quite frankly, nobody is going to bother attacking a filesystem this way - there is literally nothing that can be gained from slowing down access to a single directory in this manner. It's a DOS at best, but there are *much* easier ways of doing that and it doesn't require parent pointers or dahash collisions at all.
Quoted 7 lines
There were also millions of parent pointer xattrs, all with the same attr name and hence the same hash value here too. Doing that made the performance totally awful. Changing the hash to crc32c and then sha512 made it much harder to induce collision slowdowns on both ends like that, though sha512 added a noticeable performance hit for what it was preventing.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
So why not change the dahash for parent pointer enabled filesystems to use crc32c everywhere?
That's not difficult to do, but it'll break name obscuring in metadump unless you know of a quick way to derive B from A and maintain crc32c(A) == crc32c(B).
Yeah. So I've sort of figured out how to spoof crc32c in much the same way that obfuscate_name() does for the dahash -- fill a buffer with replacement characters, figure out which bits to flip to end up with the same crc at the end. I quickly found Mark Adler's spoof program which seems to work more reliably.
But after a bit of performance comparison, I realized that constructing namehash collisions with same-length strings is about as computationally easy with crc32c as it is with the dahash. This isn't any better than what we have now. I ran a dumb benchmark of the kernel crc32c implementation vs. dahash and noticed that for the sizes we care about (i.e. 255 bytes or less) the old dahash is only slightly slower than hardware accelerated crc32c.
I then noticed that obfuscate_name() totally does the wrong thing if ascii-ci is enabled, and indeed, trying to metadump such a filesystem causes metadump to throw errors all over the place.
In short: doing this doesn't buy us anything, but now someone has to implement a robust obfuscate_name() for crc32c and we have more software to maintain.
Quoted 10 lines
Quoted 9 lines
Not easily, but....
Quoted 2 lines
Granted I had mostly written off name obscuring in metadump on parent pointer filesystems anyway.
... so had I. The name obfuscation needs a completely different approach for PP enabled filesystems because of the name also being embedded in non-user visible structures and potentially changing attr tree dahash-based indexing.
Quoted 4 lines
<nod> I've wondered if we just have to record (dir_ino/name) -> (newname) and use that to update the parent pointer and any dabtree entries we find. That's going to be a bit of a mess to iron out since we can't change the "mounted" filesystem.
For now I've fixed this by implementing a gigantic name remapping table so that we remember (dir_ino, old_name, new_name) tuples. When we encounter the equivalent parent pointer, we can use the exact same obfuscated name, which means that xfs_repair on a restored obscured metadump works again.
While I was at it, I also implemented a 'parent' command for xfs_db that dumps the parents of any inode, similar to the existing xfs_io command.
Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
I'd much prefer we have a parent pointer index key that is fixed length, has constant hash time and a dabtree hash that is far more resilient to collision farming than to have to perform encoding/decoding of varaible length values into the key to work around a collision-rich hashing algorithm.
Wellllll I was 10 minutes away from sending v11 with all of my changes integrated, but I guess now I'll go rework the dabtree to use crc32c, encode the crc32c in the parent pointer attr name, and get rid of all the variable length value decoding. Now that I just got rid of all the intermediate patches with hash-in-attr-name.
Quoted 5 lines
I apologise for not being able to comment on this earlier. What time I've had available over the past coule of weeks has been focussed on the parts of the repair patchset that are ready for merge and I've had little time to follow the progress of the parts you are currently developing.
Quoted 6 lines
<nod> Sorry, that was my kneejerk reaction to "let's go reopen this thing and go back to where you were a week ago". I appreciate yuor help getting the online repair patchset ready for merge.
In the end it took about half an hour to put everything back, so I'll go test that overnight and let's see where we end up tomorrow.
Now that I've had a day to go sift through all of these changes, I've realized that you're right -- putting the ondisk format back to:
(parent_ino, parent_gen, namehash) -> (dirent name)
Simplifes the parent pointer ondisk conversion and runtime code quite a bit. It also makes it easy to implement obfuscated metadumps. So I'll go run v11 through fstestscloud with all of today's changes integrated, and we'll see about getting something out in a few days.
Again, sorry for being a jerk.
--D
Quoted 6 lines
--D
Quoted 4 lines
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCHSET v10r1d2 00/17] xfs: encode parent pointer name in xattr key
In reply to Allison HendersonQuoted 91 lines · Allison Henderson
Quoted 84 lines · Darrick J. Wong
Hi all,
As I've mentioned in past comments on the parent pointers patchset, the
proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:(
parent_ino,parent_gen, diroffset) -> (dirent_name)If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take
a diroffset and try to create the new entry at that offset. This will
not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping).
Requiring matching diroffsets also prevents reorganization and compaction of directories.This could be remedied by recording the parent pointer diroffset updates
necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new
directory structure. If there are a large number of diroffset updates,
then the directory commit could take an even longer time.Worse yet, if the logged xattr updates fill up the transaction, repair
will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes.
This is a completely bad engineering choice.Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(
parent_ino,dirent_name) -> NULLThe xattr code supports xattrs with zero-length values, surprisingly. The
parent_genfield makes it easy to export parent handle information,
so it can be retained:(
parent_ino,parent_gen,dirent_name) -> NULLMoving the ondisk format to this format is very advantageous for repair
code. Unfortunately, there is one hitch: xattr names cannot exceed 255
bytes due to ondisk format limitations. We don't want to constrain the
length of dirent names, so instead we create a special VLOOKUP mode for
extended attributes that allows parent pointers to require matching on
both the name and the value.The ondisk format of a parent pointer can then become:
(
parent_ino,parent_gen, dirent_name[0:242]) -> (dirent_name[243:255])Because we can always look up a specific parent pointer. Most of the patches in this patchset prepare the high level xattr code and the lower
level logging code to do this correctly, and the last patch switches the
ondisk format.Quoted 6 lines
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. kernel git tree:
Quoted 7 lines
So I gave this set a look over and for the most part I think it's ok as long as folks agree on the new format? It's a lot different from what folks originally articulated that they wanted, but for the most part the format change doesnt affect parent pointer mechanics as much as it will affect other features that may use it. I havnt seen much complaints in response, so if it's all the same to everyone else, I am fine with moving forward with it? Thanks all!
Yes, let's move forward. As I mentioned on IRC last night I'll merge
my changes into parent pointers v11 and put that out next weekish.
--D
Quoted 39 lines
Allison
Quoted 36 lines
fstests git tree:
https://urldefense.com/v3/__https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key__;!!ACWV5N9M2RV99hQ!NDopQXFVRjDfqgJvGMQRzS3fXyyd1SC4HfGl4dxSMWxNPsNKScecjp7bZYX-dDhL5tatk_xxJFdQwI5iKKgk$
---
fs/xfs/libxfs/xfs_attr.c | 66 +++++--- fs/xfs/libxfs/xfs_attr.h | 5 + fs/xfs/libxfs/xfs_attr_leaf.c | 41 ++++- fs/xfs/libxfs/xfs_da_btree.h | 6 + fs/xfs/libxfs/xfs_da_format.h | 39 ++++- fs/xfs/libxfs/xfs_fs.h | 2fs/xfs/libxfs/xfs_log_format.h| 31 +++- fs/xfs/libxfs/xfs_parent.c | 215 +++++++++++++++++---------- fs/xfs/libxfs/xfs_parent.h | 46 +++---fs/xfs/libxfs/xfs_trans_resv.c| 7 + fs/xfs/scrub/dir.c | 34 +--- fs/xfs/scrub/dir_repair.c | 87 +++-------- fs/xfs/scrub/parent.c | 48 +----- fs/xfs/scrub/parent_repair.c | 55 ++----- fs/xfs/scrub/trace.h | 65 +------- fs/xfs/xfs_attr_item.c | 318
+++++++++++++++++++++++++++++++---------
fs/xfs/xfs_attr_item.h | 3 fs/xfs/xfs_inode.c | 30 ++-- fs/xfs/xfs_ondisk.h | 1 fs/xfs/xfs_parent_utils.c | 8 + fs/xfs/xfs_symlink.c | 3 fs/xfs/xfs_xattr.c | 5 +22 files changed, 660 insertions(+), 455 deletions(-)
cover xfsprogs: tool fixes for parent pointers
In reply to Darrick J. WongHi all,
Here are a bunch of tooling changes for the parent pointers code. The only new feature here is to decode the parent pointer xattr name in xfs_db so that we can interpret (and someday fuzz) them. Everything else are bug fixes.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-toolfixes
--- db/attr.c | 31 ++++++++++++++++++++++ db/attrshort.c | 25 ++++++++++++++++++ db/metadump.c | 34 +++++++++++++++++------- include/libxfs.h | 1 + io/parent.c | 5 ++-- libxfs/init.c | 7 +++++ libxfs/libxfs_api_defs.h | 4 +++ libxfs/util.c | 14 ++++++++++ mkfs/proto.c | 65 +++++++++++++++++++++++++++++++++------------- repair/da_util.c | 2 + 10 files changed, 157 insertions(+), 31 deletions(-)
PATCH 1/9 libxfs: initialize the slab cache for parent defer items
From: Darrick J. Wong <[email protected]>
Initialize the slab cache for parent defer items. We'll need this in an upcoming patch.
--- include/libxfs.h | 1 + libxfs/init.c | 3 +++ 2 files changed, 4 insertions(+)
include/libxfs.h +1 -0
@@ -78,6 +78,7 @@ struct iomap;#include "xfs_refcount_btree.h" #include "xfs_refcount.h" #include "xfs_btree_staging.h" +#include "xfs_parent.h" #ifndef ARRAY_SIZE #define ARRAY_SIZE (x) (sizeof(x) / sizeof((x)[0]))
libxfs/init.c +3 -0
@@ -258,6 +258,8 @@ init_caches(void) "xfs_extfree_item"); xfs_trans_cache =kmem_cache_init (sizeof (struct xfs_trans ),"xfs_trans" ); + xfs_parent_intent_cache =kmem_cache_init ( +sizeof (struct xfs_parent_defer ),"xfs_parent_defer" ); } static int
@@ -275,6 +277,7 @@ destroy_caches(void)xfs_btree_destroy_cur_caches (); leaked +=kmem_cache_destroy (xfs_extfree_item_cache); leaked +=kmem_cache_destroy (xfs_trans_cache); + leaked +=kmem_cache_destroy (xfs_parent_intent_cache);return leaked; }
PATCH 2/9 mkfs: fix libxfs api misuse
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix libxfs usage problems as pointed out by xfs/437.
--- libxfs/libxfs_api_defs.h | 1 + mkfs/proto.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)
libxfs/libxfs_api_defs.h +1 -0
@@ -124,6 +124,7 @@#define xfs_initialize_perag libxfs_initialize_perag#define xfs_initialize_perag_data libxfs_initialize_perag_data#define xfs_init_local_fork libxfs_init_local_fork +#define xfs_init_parent_name_rec libxfs_init_parent_name_rec#define xfs_inobt_maxrecs libxfs_inobt_maxrecs#define xfs_inobt_stage_cursor libxfs_inobt_stage_cursor
mkfs/proto.c +2 -2
@@ -613,8 +613,8 @@ parseproto( .value = (void *)xname.name, .valuelen = xname.len, }; -xfs_init_parent_name_rec (&rec, pip, offset); - error =xfs_attr_set (&args); +libxfs_init_parent_name_rec (&rec, pip, offset); + error = -libxfs_attr_set (&args);if (error)fail (_ ("Error creating parent pointer" ), error); }
PATCH 3/9 libxfs: create new files with attr forks if necessary
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create new files with attr forks if they're going to have parent pointers. In the next patch we'll fix mkfs to use the same parent creation functions as the kernel, so we're going to need this.
--- libxfs/init.c | 4 ++++ libxfs/util.c | 14 ++++++++++++++ 2 files changed, 18 insertions(+)
libxfs/init.c +4 -0
@@ -746,14 +746,18 @@ voidlibxfs_compute_all_maxlevels ( struct xfs_mount *mp) { +struct xfs_ino_geometry *igeo =M_IGEO (mp); +xfs_alloc_compute_maxlevels (mp);xfs_bmap_compute_maxlevels (mp,XFS_DATA_FORK );xfs_bmap_compute_maxlevels (mp,XFS_ATTR_FORK ); + igeo->attr_fork_offset =xfs_bmap_compute_attr_offset (mp);xfs_ialloc_setup_geometry (mp);xfs_rmapbt_compute_maxlevels (mp);xfs_refcountbt_compute_maxlevels (mp);xfs_agbtree_compute_maxlevels (mp); + }/*
libxfs/util.c +14 -0
@@ -322,6 +322,20 @@ libxfs_init_new_inode(ASSERT (0 ); } +/* +* If we need to create attributes immediately after allocating the +* inode, initialise an empty attribute fork right now. We use the +* default fork offset for attributes here as we don't know exactly what +* size or how many attributes we might be adding. We can do this +* safely here because we know the data fork is completely empty and +* this saves us from needing to run a separate transaction to set the +* fork offset in the immediate future. +*/ +if (xfs_has_parent (tp->t_mountp) &&xfs_has_attr (tp->t_mountp)) { + ip->i_forkoff =xfs_default_attroffset (ip) >>3 ; +xfs_ifork_init_attr (ip,XFS_DINODE_FMT_EXTENTS ,0 ); + } +/* * Log the new values stuffed into the inode. */
PATCH 4/9 mkfs: fix subdir parent pointer creation
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Rework the protofile code so that it uses the same deferred parent pointer ops that the kernel uses to create parent pointers. While we're at it, make it so that subdirs of the root directory and reserved files also get parent pointers. Found by xfs/019.
--- libxfs/libxfs_api_defs.h | 3 ++ mkfs/proto.c | 65 +++++++++++++++++++++++++++++++++------------- 2 files changed, 50 insertions(+), 18 deletions(-)
libxfs/libxfs_api_defs.h +3 -0
@@ -140,6 +140,9 @@#define xfs_log_get_max_trans_res libxfs_log_get_max_trans_res#define xfs_log_sb libxfs_log_sb#define xfs_mode_to_ftype libxfs_mode_to_ftype +#define xfs_parent_defer_add libxfs_parent_defer_add +#define xfs_parent_finish libxfs_parent_finish +#define xfs_parent_start libxfs_parent_start#define xfs_perag_get libxfs_perag_get#define xfs_perag_put libxfs_perag_put#define xfs_prealloc_blocks libxfs_prealloc_blocks
mkfs/proto.c +47 -18
155 lines not on this page · raw message
PATCH 5/9 xfs_db: report parent pointer keys
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Decode the parent pointer inode, generation, and diroffset fields.
--- db/attr.c | 31 +++++++++++++++++++++++++++++++ db/attrshort.c | 25 +++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
db/attr.c +31 -0
@@ -19,6 +19,7 @@ static int attr_leaf_entries_count(void *obj, int startoff);static int attr_leaf_hdr_count (void *obj,int startoff);static int attr_leaf_name_local_count (void *obj,int startoff);static int attr_leaf_name_local_name_count (void *obj,int startoff); +static int attr_leaf_name_pptr_count (void *obj,int startoff);static int attr_leaf_name_local_value_count (void *obj,int startoff);static int attr_leaf_name_local_value_offset (void *obj,int startoff,int idx);
@@ -111,6 +112,8 @@ const field_t attr_leaf_map_flds[] = {#define LNOFF (f) bitize(offsetof(xfs_attr_leaf_name_local_t, f))#define LVOFF (f) bitize(offsetof(xfs_attr_leaf_name_remote_t, f)) +#define PPOFF (f) bitize(offsetof(xfs_attr_leaf_name_local_t, nameval) + \ + offsetof(struct xfs_parent_name_rec, f))const field_t attr_leaf_name_flds[] = { {"valuelen" ,FLDT_UINT16D ,OI (LNOFF (valuelen)), attr_leaf_name_local_count,FLD_COUNT ,TYP_NONE },
@@ -118,6 +121,12 @@ const field_t attr_leaf_name_flds[] = { attr_leaf_name_local_count, FLD_COUNT, TYP_NONE }, {"name" ,FLDT_CHARNS ,OI (LNOFF (nameval)), attr_leaf_name_local_name_count, FLD_COUNT, TYP_NONE }, + { "parent_ino", FLDT_INO, OI(PPOFF(p_ino)), + attr_leaf_name_pptr_count, FLD_COUNT, TYP_INODE }, + { "parent_gen", FLDT_UINT32D, OI(PPOFF(p_gen)), + attr_leaf_name_pptr_count, FLD_COUNT, TYP_NONE }, + { "parent_diroffset", FLDT_UINT32D, OI(PPOFF(p_diroffset)), + attr_leaf_name_pptr_count, FLD_COUNT, TYP_NONE }, { "value", FLDT_CHARNS, attr_leaf_name_local_value_offset, attr_leaf_name_local_value_count, FLD_COUNT|FLD_OFFSET, TYP_NONE }, { "valueblk", FLDT_UINT32X, OI(LVOFF(valueblk)),
@@ -273,6 +282,26 @@ attr_leaf_name_local_count( __attr_leaf_name_local_count); } +static int +__attr_leaf_name_pptr_count ( +struct xfs_attr_leafblock *leaf, +struct xfs_attr_leaf_entry *e, +int i) +{ +if (e->flags &XFS_ATTR_PARENT ) +return 1 ; +return 0 ; +} + +static int +attr_leaf_name_pptr_count ( +void *obj, +int startoff) +{ +return attr_leaf_entry_walk (obj, startoff, + __attr_leaf_name_pptr_count); +} + static int __attr_leaf_name_local_name_count( struct xfs_attr_leafblock *leaf,
@@ -283,6 +312,8 @@ __attr_leaf_name_local_name_count(if (!(e->flags &XFS_ATTR_LOCAL ))return 0 ; +if (e->flags &XFS_ATTR_PARENT ) +return 0 ; l =xfs_attr3_leaf_name_local (leaf, i);return l->namelen;
db/attrshort.c +25 -0
@@ -13,6 +13,7 @@#include "attrshort.h" static int attr_sf_entry_name_count (void *obj,int startoff); +static int attr_sf_entry_pptr_count (void *obj,int startoff);static int attr_sf_entry_value_count (void *obj,int startoff);static int attr_sf_entry_value_offset (void *obj,int startoff,int idx);static int attr_shortform_list_count (void *obj,int startoff);
@@ -34,6 +35,8 @@ const field_t attr_sf_hdr_flds[] = { };#define EOFF (f) bitize(offsetof(struct xfs_attr_sf_entry, f)) +#define PPOFF (f) bitize(offsetof(struct xfs_attr_sf_entry, nameval) + \ + offsetof(struct xfs_parent_name_rec, f))const field_t attr_sf_entry_flds[] = { {"namelen" ,FLDT_UINT8D ,OI (EOFF (namelen)),C1 ,0 ,TYP_NONE }, {"valuelen" ,FLDT_UINT8D ,OI (EOFF (valuelen)),C1 ,0 ,TYP_NONE },
@@ -49,11 +52,31 @@ const field_t attr_sf_entry_flds[] = { TYP_NONE }, {"name" ,FLDT_CHARNS ,OI (EOFF (nameval)), attr_sf_entry_name_count, FLD_COUNT, TYP_NONE }, + { "parent_ino", FLDT_INO, OI(PPOFF(p_ino)), attr_sf_entry_pptr_count, + FLD_COUNT, TYP_INODE }, + { "parent_gen", FLDT_UINT32D, OI(PPOFF(p_gen)), attr_sf_entry_pptr_count, + FLD_COUNT, TYP_NONE }, + { "parent_diroffset", FLDT_UINT32D, OI(PPOFF(p_diroffset)), + attr_sf_entry_pptr_count, FLD_COUNT, TYP_NONE }, { "value", FLDT_CHARNS, attr_sf_entry_value_offset, attr_sf_entry_value_count, FLD_COUNT|FLD_OFFSET, TYP_NONE }, { NULL } }; +static int +attr_sf_entry_pptr_count ( +void *obj, +int startoff) +{ +struct xfs_attr_sf_entry *e; + +ASSERT (bitoffs (startoff) ==0 ); + e = (struct xfs_attr_sf_entry *)((char *)obj +byteize (startoff)); +if (e->flags &XFS_ATTR_PARENT ) +return 1 ; +return 0 ; +} + static int attr_sf_entry_name_count( void *obj,
@@ -63,6 +86,8 @@ attr_sf_entry_name_count(ASSERT (bitoffs (startoff) ==0 ); e = (struct xfs_attr_sf_entry *)((char *)obj +byteize (startoff)); +if (e->flags &XFS_ATTR_PARENT ) +return 0 ;return e->namelen; }
PATCH 6/9 xfs_db: obfuscate dirent and pptr names consistently
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
When someone wants to perform an obfuscated metadump of a filesystem where parent pointers are enabled, we have to use the *exact* same obfuscated name for both the directory entry and the parent pointer. Instead of using an RNG to influence the obfuscated name, use the dirent inode number to start the obfuscated name. This makes them consistent, though the resulting names aren't quite so full of control characters.
--- db/metadump.c | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-)
db/metadump.c +24 -10
@@ -740,12 +740,14 @@ nametable_add(xfs_dahash_t hash, int namelen, unsigned char *name)#define rol32 (x,y) (((x) << (y)) | ((x) >> (32 - (y))))static inline unsigned char -random_filename_char (void ) +random_filename_char (xfs_ino_t ino) {static unsigned char filename_alphabet[] ="ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789-_" ; +if (ino) +return filename_alphabet[ino % (sizeof filename_alphabet -1 )];return filename_alphabet[random () % (sizeof filename_alphabet -1 )]; }
@@ -815,6 +817,7 @@ in_lost_found(*/ static void obfuscate_name ( +xfs_ino_t ino,xfs_dahash_t hash,size_t name_len,unsigned char *name)
@@ -842,7 +845,7 @@ obfuscate_name(* Accumulate its new hash value as we go. */ for (i =0 ; i < name_len -5 ; i++) { - *newp =random_filename_char (); + *newp =random_filename_char (ino); new_hash = *newp ^rol32 (new_hash,7 ); newp++; }
@@ -1207,7 +1210,10 @@ generate_obfuscated_name(/* Obfuscate the name (if possible) */ hash =libxfs_da_hashname (name, namelen); -obfuscate_name (hash, namelen, name); +if (xfs_has_parent (mp)) +obfuscate_name (ino, hash, namelen, name); +else +obfuscate_name (0 , hash, namelen, name);/* * Make sure the name is not something already seen. If we
@@ -1320,7 +1326,7 @@ obfuscate_path_components(/* last (or single) component */ namelen =strnlen ((char *)comp, len); hash =libxfs_da_hashname (comp, namelen); -obfuscate_name (hash, namelen, comp); +obfuscate_name (0 , hash, namelen, comp);break ; } namelen = slash - (char *)comp;
@@ -1331,7 +1337,7 @@ obfuscate_path_components(continue ; } hash =libxfs_da_hashname (comp, namelen); -obfuscate_name (hash, namelen, comp); +obfuscate_name (0 , hash, namelen, comp); comp += namelen +1 ; len -= namelen +1 ; }
@@ -1407,10 +1413,15 @@ process_sf_attr( }if (obfuscate) { -generate_obfuscated_name (0 , asfep->namelen, - &asfep->nameval[0 ]); -memset (&asfep->nameval[asfep->namelen],'v' , - asfep->valuelen); +if (asfep->flags &XFS_ATTR_PARENT ) { +generate_obfuscated_name (cur_ino, asfep->valuelen, + &asfep->nameval[asfep->namelen]); + }else { +generate_obfuscated_name (0 , asfep->namelen, + &asfep->nameval[0 ]); +memset (&asfep->nameval[asfep->namelen],'v' , + asfep->valuelen); + } } asfep = (struct xfs_attr_sf_entry *)((char *)asfep +
@@ -1785,7 +1796,7 @@ process_attr_block( (long long)cur_ino);break ; } -if (obfuscate) { +if (obfuscate && !(entry->flags &XFS_ATTR_PARENT )) {generate_obfuscated_name (0 , local->namelen, &local->nameval[0 ]); memset(&local->nameval[local->namelen], 'v',
@@ -1797,6 +1808,9 @@ process_attr_block( zlen =xfs_attr_leaf_entsize_local (nlen, vlen) - (sizeof (xfs_attr_leaf_name_local_t) -1 + nlen + vlen); +if (obfuscate && (entry->flags &XFS_ATTR_PARENT )) +generate_obfuscated_name (cur_ino, vlen, + &local->nameval[nlen]);if (zero_stale_data)memset (&local->nameval[nlen + vlen],0 , zlen); }else {
PATCH 7/9 xfs_io: print path in path_print
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Actually print the path string once we've bothered to construct it into a string buffer.
--- io/parent.c | 2 ++ 1 file changed, 2 insertions(+)
io/parent.c +2 -0
@@ -87,6 +87,8 @@ path_print( ret =path_list_to_string (path, buf + ret, len - ret);if (ret <0 )return ret; + +printf ("%s\n" , buf);return 0 ; }
PATCH 8/9 xfs_io: parent command is not experts-only
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
This command isn't dangerous, so don't make it experts-only.
--- io/parent.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
io/parent.c +1 -2
@@ -250,6 +250,5 @@ parent_init(void) parent_cmd.oneline =_ ("print parent inodes" ); parent_cmd.help = parent_help; -if (expert) -add_command (&parent_cmd); +add_command (&parent_cmd); }
PATCH 9/9 xfs_repair: fix incorrect dabtree hashval comparison
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If an xattr structure contains enough names with the same hash value to fill multiple xattr leaf blocks with names all hashing to the same value, then the dabtree nodes will contain consecutive entries with the same hash value.
This causes false corruption reports in xfs_repair because it's not expecting such a huge same-hashing structure. Fix that.
--- repair/da_util.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
repair/da_util.c +1 -1
@@ -330,7 +330,7 @@ _("%s block used/count inconsistency - %d/%hu\n"),/* * hash values monotonically increasing ??? */ -if (cursor->level[this_level].hashval >= +if (cursor->level[this_level].hashval >be32_to_cpu (nodehdr.btree[entry].hashval)) { do_warn( _("%s block hashvalue inconsistency, expected > %u / saw %u\n"),
cover xfsprogs: actually use getparent ioctl
In reply to Darrick J. WongHi all,
Yikes. The userspace parent pointers support code dates from 2017 and is very very moldy. This patchset moves the xfs_io filtering stuff back to xfs_io. It also moves the parent pointer support code to libfrog because we don't want to expose things via libhandle until we're absolutely sure that we want to do that.
(We probably want to do that some day.)
Finally, adapt xfs_scrub to use parent pointer information whenever it has something to say about a file handle that it has open.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-use-getparents
--- scrub/common.c | 21 +++++++++++++++++++++ scrub/inodes.c | 26 -------------------------- scrub/inodes.h | 2 -- 3 files changed, 21 insertions(+), 28 deletions(-)
PATCH 1/2 xfs_scrub: revert unnecessary code from "implement the upper half of parent pointers"
From: Darrick J. Wong <[email protected]>
Revert this piece which is no longer necessary.
--- scrub/inodes.c | 26 -------------------------- scrub/inodes.h | 2 -- 2 files changed, 28 deletions(-)
scrub/inodes.c +0 -26
@@ -19,7 +19,6 @@#include "descr.h" #include "libfrog/fsgeom.h" #include "libfrog/bulkstat.h" -#include "parent.h" /* * Iterate a range of inodes.
@@ -450,28 +449,3 @@ scrub_open_handle(return open_by_fshandle (handle,sizeof (*handle),O_RDONLY |O_NOATIME |O_NOFOLLOW |O_NOCTTY ); } - -/* Construct a description for an inode. */ -void -xfs_scrub_ino_descr ( -struct scrub_ctx *ctx, -struct xfs_handle *handle, -char *buf, -size_t buflen) -{ -uint64_t ino; -xfs_agnumber_t agno; -xfs_agino_t agino; -int ret; - - ret =handle_to_path (handle,sizeof (struct xfs_handle ), buf, buflen); -if (ret >=0 ) -return ; - - ino = handle->ha_fid.fid_ino; - agno = ino / (1ULL << (ctx->mnt.inopblog + ctx->mnt.agblklog)); - agino = ino % (1ULL << (ctx->mnt.inopblog + ctx->mnt.agblklog)); -snprintf (buf, buflen,_ ("inode %" PRIu64" (%u/%u)" ), ino, agno, - agino); -} -
scrub/inodes.h +0 -2
@@ -21,7 +21,5 @@ int scrub_scan_all_inodes(struct scrub_ctx *ctx, scrub_inode_iter_fn fn, void *arg);int scrub_open_handle (struct xfs_handle *handle); -void xfs_scrub_ino_descr (struct scrub_ctx *ctx,struct xfs_handle *handle, -char *buf,size_t buflen);#endif /* XFS_SCRUB_INODES_H_ */
PATCH 2/2 xfs_scrub: use parent pointers when possible to report file operations
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
If parent pointers are available, use them to supply file paths when doing things to files, instead of merely printing the inode number.
--- scrub/common.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
scrub/common.c +21 -0
@@ -12,6 +12,7 @@#include "xfs_scrub.h" #include "common.h" #include "progress.h" +#include "libfrog/pptrs.h" extern char *progname;
@@ -407,6 +408,26 @@ scrub_render_ino_descr(uint32_t agino;int ret; +if (ctx->mnt.fsgeom.flags &XFS_FSOP_GEOM_FLAGS_PARENT ) { +struct xfs_handle handle; + +memcpy (&handle.ha_fsid, ctx->fshandle,sizeof (handle.ha_fsid)); + handle.ha_fid.fid_len =sizeof (xfs_fid_t) - +sizeof (handle.ha_fid.fid_len); + handle.ha_fid.fid_pad =0 ; + handle.ha_fid.fid_ino = ino; + handle.ha_fid.fid_gen = gen; + + ret =handle_to_path (&handle,sizeof (struct xfs_handle ), buf, + buflen); +/* +* If successful, return any positive integer to use the +* formatted error string. +*/ +if (ret ==0 ) +return 1 ; + } + agno =cvt_ino_to_agno (&ctx->mnt, ino); agino =cvt_ino_to_agino (&ctx->mnt, ino); ret = snprintf(buf, buflen, _("inode %"PRIu64" (%"PRIu32"/%"PRIu32")%s"),
cover libfrog: fix parent pointer library code
In reply to Darrick J. WongHi all,
This series moves the parent pointer library code out of libhandle and into libfrog. This is necessary because libhandle is exported as a userspace library, and we don't want to export the parent pointer stuff until we're absolutely ready to do that. So that move is made in the first patch.
The rest of the patchset fixes various bugs and inconsistencies and bitrot that have cropped up since I wrote this code in 2017.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-fix-libfrog-code
--- include/parent.h | 25 ------ io/parent.c | 117 +++++++++++++++++----------- libfrog/Makefile | 2 libfrog/paths.c | 46 +++++++++-- libfrog/paths.h | 8 +- libfrog/pptrs.c | 217 ++++++++++++++++++++++------------------------------ libfrog/pptrs.h | 25 ++++++ libhandle/Makefile | 2 8 files changed, 237 insertions(+), 205 deletions(-) rename libhandle/parent.c => libfrog/pptrs.c (50%) create mode 100644 libfrog/pptrs.h
PATCH 1/7 xfs_io: move parent pointer filtering and formatting flags out of libhandle
From: Darrick J. Wong <[email protected]>
All this filtering and presentation stuff originates in xfs_io and should stay there. The added arguments seriously complicate the basic iterator interface and there are no other users.
While we're at it, fix a bug in path_print where it doesn't actually print the path.
--- include/parent.h | 17 +++------ io/parent.c | 99 ++++++++++++++++++++++++++++++++-------------------- libfrog/paths.c | 28 ++++++++++++++- libfrog/paths.h | 8 +++- libhandle/parent.c | 77 ++++++++++++---------------------------- 5 files changed, 119 insertions(+), 110 deletions(-)
include/parent.h +5 -12
@@ -17,27 +17,20 @@ typedef struct parent_cursor {__u32 opaque[4 ];/* an opaque cookie */ } parent_cursor_t; -/* Print parent pointer option flags */ -#define XFS_PPPTR_OFLAG_SHORT (1<<0)/* Print in short format */ -struct path_list ;typedef int (*walk_pptr_fn )(struct xfs_pptr_info *pi,struct xfs_parent_ptr *pptr, -void *arg,int flags); +void *arg);typedef int (*walk_ppath_fn )(const char *mntpt,struct path_list *path,void *arg);#define WALK_PPTRS_ABORT 1 -int fd_walk_pptrs (int fd,uint64_t pino,char *pname,walk_pptr_fn fn, -void *arg,int flags); -int handle_walk_pptrs (void *hanp,size_t hanlen,uint64_t pino,char *pname, -walk_pptr_fn fn,void *arg,int flags); +int fd_walk_pptrs (int fd,walk_pptr_fn fn,void *arg); +int handle_walk_pptrs (void *hanp,size_t hanlen,walk_pptr_fn fn,void *arg);#define WALK_PPATHS_ABORT 1 -int fd_walk_ppaths (int fd,uint64_t pino,char *pname,walk_ppath_fn fn, -void *arg,int flags); -int handle_walk_ppaths (void *hanp,size_t hanlen,uint64_t pino,char *pname, -walk_ppath_fn fn,void *arg,int flags); +int fd_walk_ppaths (int fd,walk_ppath_fn fn,void *arg); +int handle_walk_ppaths (void *hanp,size_t hanlen,walk_ppath_fn fn,void *arg);int fd_to_path (int fd,char *path,size_t pathlen);int handle_to_path (void *hanp,size_t hlen,char *path,size_t pathlen);
io/parent.c +60 -39
200 lines not on this page · raw message
libfrog/paths.c +26 -2
@@ -574,13 +574,15 @@ struct path_list {struct path_component {struct list_head pc_list; +uint64_t pc_ino;char *pc_fname; };/* Initialize a path component with a given name. */ struct path_component *path_component_init ( -const char *name) +const char *name, +uint64_t ino) {struct path_component *pc;
@@ -593,6 +595,7 @@ path_component_init(free (pc);return NULL ; } + pc->pc_ino = ino;return pc; }
@@ -610,7 +613,8 @@ intpath_component_change ( struct path_component *pc, void *name, - size_t namelen) + size_t namelen, + uint64_t ino) { void *p;
@@ -620,6 +624,7 @@ path_component_change( pc->pc_fname = p;memcpy (pc->pc_fname, name, namelen); pc->pc_fname[namelen] =0 ; + pc->pc_ino = ino;return 0 ; }
@@ -699,3 +704,22 @@ path_list_to_string( }return bytes; } + +/* Walk each component of a path. */ +int +path_walk_components ( +struct path_list *path, +path_walk_fn_t fn, +void *arg) +{ +struct path_component *pos; +int ret; + +list_for_each_entry (pos, &path->p_head, pc_list) { + ret =fn (pos->pc_fname, pos->pc_ino, arg); +if (ret) +return ret; + } + +return 0 ; +}
libfrog/paths.h +6 -2
@@ -63,10 +63,10 @@ extern fs_path_t *fs_cursor_next_entry(fs_cursor_t *__cp);struct path_list ;struct path_component ; -struct path_component *path_component_init (const char *name); +struct path_component *path_component_init (const char *name,uint64_t ino);void path_component_free (struct path_component *pc);int path_component_change (struct path_component *pc,void *name, -size_t namelen); +size_t namelen,uint64_t ino);struct path_list *path_list_init (void );void path_list_free (struct path_list *path);
@@ -77,4 +77,8 @@ void path_list_del_component(struct path_list *path, struct path_component *pc);ssize_t path_list_to_string (struct path_list *path,char *buf,size_t buflen); +typedef int (*path_walk_fn_t )(const char *name,uint64_t ino,void *arg); + +int path_walk_components (struct path_list *path,path_walk_fn_t fn,void *arg); +#endif /* __PATH_H__ */
libhandle/parent.c +22 -55
196 lines not on this page · raw message
PATCH 2/7 libfrog: remove all the parent pointer code from libhandle
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Move this code out of libhandle and into libfrog. We don't want to expose this stuff to a userspace library until customers actually demand it. While we're here, fix the copyright statements and licensing tags.
--- include/parent.h | 18 ------------------ io/parent.c | 1 + libfrog/Makefile | 2 ++ libfrog/pptrs.c | 22 ++++------------------ libfrog/pptrs.h | 27 +++++++++++++++++++++++++++ libhandle/Makefile | 2 +- 6 files changed, 35 insertions(+), 37 deletions(-) rename libhandle/parent.c => libfrog/pptrs.c (87%) create mode 100644 libfrog/pptrs.h
include/parent.h +0 -18
@@ -17,22 +17,4 @@ typedef struct parent_cursor { __u32 opaque[4]; /* an opaque cookie */ } parent_cursor_t; -struct path_list ; - -typedef int (*walk_pptr_fn )(struct xfs_pptr_info *pi,struct xfs_parent_ptr *pptr, -void *arg); -typedef int (*walk_ppath_fn )(const char *mntpt,struct path_list *path, -void *arg); - -#define WALK_PPTRS_ABORT 1 -int fd_walk_pptrs (int fd,walk_pptr_fn fn,void *arg); -int handle_walk_pptrs (void *hanp,size_t hanlen,walk_pptr_fn fn,void *arg); - -#define WALK_PPATHS_ABORT 1 -int fd_walk_ppaths (int fd,walk_ppath_fn fn,void *arg); -int handle_walk_ppaths (void *hanp,size_t hanlen,walk_ppath_fn fn,void *arg); - -int fd_to_path (int fd,char *path,size_t pathlen); -int handle_to_path (void *hanp,size_t hlen,char *path,size_t pathlen); -#endif
io/parent.c +1 -0
@@ -8,6 +8,7 @@#include "input.h" #include "libfrog/paths.h" #include "parent.h" +#include "libfrog/pptrs.h" #include "handle.h" #include "init.h" #include "io.h"
libfrog/Makefile +2 -0
@@ -23,6 +23,7 @@ list_sort.c \ linux.c \ logging.c \ paths.c \ +pptrs.c \ projects.c \ ptvar.c \ radix-tree.c \
@@ -42,6 +43,7 @@ crc32table.h \ fsgeom.h \ logging.h \ paths.h \ +pptrs.h \ projects.h \ ptvar.h \ radix-tree.h \
libfrog/pptrs.c renamed from libhandle/parent.c +4 -18
@@ -1,21 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0-or-later /* -* Copyright (C) 2017 Oracle. All Rights Reserved. -* -* Author: Darrick J. Wong <[email protected]> -* -* This program is free software; you can redistribute it and/or -* modify it under the terms of the GNU General Public License -* as published by the Free Software Foundation; either version 2 -* of the License, or (at your option) any later version. -* -* This program is distributed in the hope that it would be useful, -* but WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -* GNU General Public License for more details. -* -* You should have received a copy of the GNU General Public License -* along with this program; if not, write the Free Software Foundation, -* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. +* Copyright (C) 2017-2023 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> */ #include "platform_defs.h" #include "xfs.h"
@@ -23,7 +9,7 @@#include "list.h" #include "libfrog/paths.h" #include "handle.h" -#include "parent.h" +#include "libfrog/pptrs.h" /* Allocate a buffer large enough for some parent pointer records. */ static inline struct xfs_pptr_info *
libfrog/pptrs.h new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Copyright (C) 2023 Oracle. All Rights Reserved. +* Author: Darrick J. Wong <[email protected]> +*/ +#ifndef __LIBFROG_PPTRS_H_ +#define __LIBFROG_PPTRS_H_ + +struct path_list ; + +typedef int (*walk_pptr_fn )(struct xfs_pptr_info *pi,struct xfs_parent_ptr *pptr, +void *arg); +typedef int (*walk_ppath_fn )(const char *mntpt,struct path_list *path, +void *arg); + +#define WALK_PPTRS_ABORT 1 +int fd_walk_pptrs (int fd,walk_pptr_fn fn,void *arg); +int handle_walk_pptrs (void *hanp,size_t hanlen,walk_pptr_fn fn,void *arg); + +#define WALK_PPATHS_ABORT 1 +int fd_walk_ppaths (int fd,walk_ppath_fn fn,void *arg); +int handle_walk_ppaths (void *hanp,size_t hanlen,walk_ppath_fn fn,void *arg); + +int fd_to_path (int fd,char *path,size_t pathlen); +int handle_to_path (void *hanp,size_t hlen,char *path,size_t pathlen); + +#endif /* __LIBFROG_PPTRS_H_ */
libhandle/Makefile +1 -1
@@ -12,7 +12,7 @@ LT_AGE = 0 LTLDFLAGS += -Wl,--version-script,libhandle.sym -CFILES = handle.c jdm.c parent.c +CFILES = handle.c jdm.c LSRCFILES = libhandle.sym default: ltdepend $(LTLIBRARY)
PATCH 3/7 libfrog: fix indenting errors in xfss_pptr_alloc
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix some indenting problems, and get rid of the xfs_ prefix.
--- libfrog/pptrs.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
libfrog/pptrs.c +9 -9
@@ -13,16 +13,16 @@/* Allocate a buffer large enough for some parent pointer records. */ static inline struct xfs_pptr_info * -xfs_pptr_alloc ( -size_t bufsize) +alloc_pptr_buf ( +size_t bufsize) { -struct xfs_pptr_info *pi; +struct xfs_pptr_info *pi; - pi =calloc (bufsize,1 ); -if (!pi) -return NULL ; - pi->pi_ptrs_size = bufsize; -return pi; + pi =calloc (bufsize,1 ); +if (!pi) +return NULL ; + pi->pi_ptrs_size = bufsize; +return pi; }/* Walk all parents of the given file handle. */
@@ -38,7 +38,7 @@ handle_walk_parents(unsigned int i;ssize_t ret =-1 ; - pi =xfs_pptr_alloc (XFS_XATTR_LIST_MAX ); + pi =alloc_pptr_buf (XFS_XATTR_LIST_MAX );if (!pi)return -1 ;
PATCH 4/7 libfrog: return positive errno in pptrs.c
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make all the functions in here return 0 for success or positive errno.
--- io/parent.c | 12 +++----- libfrog/paths.c | 11 +++++-- libfrog/pptrs.c | 81 +++++++++++++++++++++++++++++++++---------------------- libfrog/pptrs.h | 6 +--- 4 files changed, 62 insertions(+), 48 deletions(-)
io/parent.c +5 -7
@@ -68,7 +68,7 @@ print_parents(else ret =fd_walk_pptrs (file->fd, pptr_print, args);if (ret) -perror (file->name); +fprintf (stderr,"%s: %s\n" , file->name,strerror (ret));return 0 ; }
@@ -106,14 +106,12 @@ path_print( } ret =snprintf (buf, len,"%s" , mntpt); -if (ret !=strlen (mntpt)) { - errno =ENOMEM ; -return -1 ; - } +if (ret !=strlen (mntpt)) +return ENAMETOOLONG ; ret =path_list_to_string (path, buf + ret, len - ret);if (ret <0 ) -return ret; +return ENAMETOOLONG ;printf ("%s\n" , buf);return 0 ;
@@ -132,7 +130,7 @@ print_paths(else ret =fd_walk_ppaths (file->fd, path_print, args);if (ret) -perror (file->name); +fprintf (stderr,"%s: %s\n" , file->name,strerror (ret));return 0 ; }
libfrog/paths.c +7 -4
@@ -608,7 +608,7 @@ path_component_free(free (pc); } -/* Change a path component's filename. */ +/* Change a path component's filename or returns positive errno. */ int path_component_change( struct path_component *pc,
@@ -620,7 +620,7 @@ path_component_change( p =realloc (pc->pc_fname, namelen +1 );if (!p) -return -1 ; +return errno; pc->pc_fname = p;memcpy (pc->pc_fname, name, namelen); pc->pc_fname[namelen] =0 ;
@@ -628,7 +628,7 @@ path_component_change(return 0 ; } -/* Initialize a pathname. */ +/* Initialize a pathname or returns positive errno. */ struct path_list *path_list_init (void ) {
@@ -683,7 +683,10 @@ path_list_del_component(list_del_init (&pc->pc_list); } -/* Convert a pathname into a string. */ +/* +* Convert a pathname into a string or returns -1 if the buffer isn't long +* enough. +*/ ssize_t path_list_to_string( struct path_list *path,
libfrog/pptrs.c +48 -33
204 lines not on this page · raw message
libfrog/pptrs.h +2 -4
@@ -8,16 +8,14 @@struct path_list ; -typedef int (*walk_pptr_fn )(struct xfs_pptr_info *pi,struct xfs_parent_ptr *pptr, -void *arg); +typedef int (*walk_pptr_fn )(struct xfs_pptr_info *pi, +struct xfs_parent_ptr *pptr,void *arg);typedef int (*walk_ppath_fn )(const char *mntpt,struct path_list *path,void *arg); -#define WALK_PPTRS_ABORT 1int fd_walk_pptrs (int fd,walk_pptr_fn fn,void *arg);int handle_walk_pptrs (void *hanp,size_t hanlen,walk_pptr_fn fn,void *arg); -#define WALK_PPATHS_ABORT 1int fd_walk_ppaths (int fd,walk_ppath_fn fn,void *arg);int handle_walk_ppaths (void *hanp,size_t hanlen,walk_ppath_fn fn,void *arg);
PATCH 5/7 libfrog: only walk one parent pointer at a time in handle_walk_parent_path_ptr
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
handle_walk_parents already walks each returned parent pointer record, so we don't need a loop in handle_walk_parent_path_ptr.
--- libfrog/pptrs.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)
libfrog/pptrs.c +10 -14
@@ -133,26 +133,22 @@ handle_walk_parent_path_ptr( {struct walk_ppath_level_info *wpli = arg;struct walk_ppaths_info *wpi = wpli->wpi; -unsigned int i;int ret =0 ;if (pi->pi_flags &XFS_PPTR_OFLAG_ROOT )return wpi->fn (wpi->mntpt, wpi->path, wpi->arg); -for (i =0 ; i < pi->pi_count; i++) { - p =xfs_ppinfo_to_pp (pi, i); - ret =path_component_change (wpli->pc, p->xpp_name, + ret =path_component_change (wpli->pc, p->xpp_name,strlen ((char *)p->xpp_name), p->xpp_ino); -if (ret) -break ; - wpli->newhandle.ha_fid.fid_ino = p->xpp_ino; - wpli->newhandle.ha_fid.fid_gen = p->xpp_gen; -path_list_add_parent_component (wpi->path, wpli->pc); - ret =handle_walk_parent_paths (wpi, &wpli->newhandle); -path_list_del_component (wpi->path, wpli->pc); -if (ret) -break ; - } +if (ret) +return ret; + + wpli->newhandle.ha_fid.fid_ino = p->xpp_ino; + wpli->newhandle.ha_fid.fid_gen = p->xpp_gen; + +path_list_add_parent_component (wpi->path, wpli->pc); + ret =handle_walk_parent_paths (wpi, &wpli->newhandle); +path_list_del_component (wpi->path, wpli->pc);return ret; }
PATCH 6/7 libfrog: trim trailing slashes when printing pptr paths
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Trim the trailing slashes in the mountpoint string when we're printing parent pointer paths.
--- io/parent.c | 9 +++++++-- libfrog/pptrs.c | 9 +++++++-- 2 files changed, 14 insertions(+), 4 deletions(-)
io/parent.c +7 -2
@@ -97,6 +97,7 @@ path_print(struct pptr_args *args = arg;char buf[PATH_MAX ];size_t len =PATH_MAX ; +int mntpt_len =strlen (mntpt);int ret;if (args->filter_ino || args->filter_name) {
@@ -105,8 +106,12 @@ path_print(return 0 ; } - ret =snprintf (buf, len,"%s" , mntpt); -if (ret !=strlen (mntpt)) +/* Trim trailing slashes from the mountpoint */ +while (mntpt_len >0 && mntpt[mntpt_len -1 ] =='/' ) + mntpt_len--; + + ret =snprintf (buf, len,"%.*s" , mntpt_len, mntpt); +if (ret != mntpt_len)return ENAMETOOLONG ; ret =path_list_to_string (path, buf + ret, len - ret);
libfrog/pptrs.c +7 -2
@@ -267,10 +267,15 @@ handle_to_path_walk( void *arg) {struct path_walk_info *pwi = arg; +int mntpt_len =strlen (mntpt);int ret; - ret =snprintf (pwi->buf, pwi->len,"%s" , mntpt); -if (ret !=strlen (mntpt)) +/* Trim trailing slashes from the mountpoint */ +while (mntpt_len >0 && mntpt[mntpt_len -1 ] =='/' ) + mntpt_len--; + + ret =snprintf (pwi->buf, pwi->len,"%.*s" , mntpt_len, mntpt); +if (ret != mntpt_len)return ENAMETOOLONG ; ret =path_list_to_string (path, pwi->buf + ret, pwi->len - ret);
PATCH 7/7 libfrog: fix a buffer overrun in path_list_to_string
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix a buffer overrun when converting a path list to a string.
--- libfrog/paths.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
libfrog/paths.c +6 -1
@@ -694,13 +694,18 @@ path_list_to_string( size_t buflen) {struct path_component *pos; +char *buf_end = buf + buflen;ssize_t bytes =0 ;int ret;list_for_each_entry (pos, &path->p_head, pc_list) { +if (buf >= buf_end) +return -1 ; + ret =snprintf (buf, buflen,"/%s" , pos->pc_fname); -if (ret !=1 +strlen (pos->pc_fname)) +if (ret <0 || ret >= buflen)return -1 ; + bytes += ret; buf += ret; buflen -= ret;
cover xfsprogs: bug fixes for parent pointers
In reply to Darrick J. WongHi all,
This series contains the accumulated bug fixes from Darrick to make fstests pass and online repair work.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-bugfixes
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-bugfixes
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-bugfixes
--- io/parent.c | 22 +++++++-------- libfrog/pptrs.c | 36 ++++++++++++------------ libfrog/pptrs.h | 4 +-- libxfs/libxfs_api_defs.h | 3 +- libxfs/xfs_da_format.h | 11 +++++++ libxfs/xfs_fs.h | 69 +++++++++++++++++++++------------------------- libxfs/xfs_parent.c | 47 +++++++++++++++++++++++++++---- libxfs/xfs_parent.h | 24 +++++++--------- mkfs/proto.c | 12 ++++---- 9 files changed, 131 insertions(+), 97 deletions(-)
PATCH 1/5 xfs: rename xfs_pptr_info to xfs_getparents
From: Darrick J. Wong <[email protected]>
Rename the head structure of the parent pointer ioctl to match the name of the ioctl (XFS_IOC_GETPARENTS).
--- io/parent.c | 4 ++-- libfrog/pptrs.c | 24 ++++++++++++------------ libfrog/pptrs.h | 2 +- libxfs/xfs_fs.h | 45 +++++++++++++++++++++++---------------------- 4 files changed, 38 insertions(+), 37 deletions(-)
io/parent.c +2 -2
@@ -24,14 +24,14 @@ struct pptr_args {static int pptr_print ( -struct xfs_pptr_info *pi, +struct xfs_getparents *pi,struct xfs_parent_ptr *pptr,void *arg) {struct pptr_args *args = arg;unsigned int namelen; -if (pi->pi_flags &XFS_PPTR_OFLAG_ROOT ) { +if (pi->gp_flags &XFS_GETPARENTS_OFLAG_ROOT ) {printf (_ ("Root directory.\n" ));return 0 ; }
libfrog/pptrs.c +12 -12
@@ -12,16 +12,16 @@#include "libfrog/pptrs.h" /* Allocate a buffer large enough for some parent pointer records. */ -static inline struct xfs_pptr_info * +static inline struct xfs_getparents *alloc_pptr_buf (size_t bufsize) { -struct xfs_pptr_info *pi; +struct xfs_getparents *pi; pi =calloc (bufsize,1 );if (!pi)return NULL ; - pi->pi_ptrs_size = bufsize; + pi->gp_ptrs_size = bufsize;return pi; }
@@ -36,7 +36,7 @@ handle_walk_parents(walk_pptr_fn fn, void *arg) { - struct xfs_pptr_info *pi; + struct xfs_getparents *pi;struct xfs_parent_ptr *p;unsigned int i;ssize_t ret =-1 ;
@@ -46,25 +46,25 @@ handle_walk_parents(return errno;if (handle) { -memcpy (&pi->pi_handle, handle,sizeof (struct xfs_handle )); - pi->pi_flags =XFS_PPTR_IFLAG_HANDLE ; +memcpy (&pi->gp_handle, handle,sizeof (struct xfs_handle )); + pi->gp_flags =XFS_GETPARENTS_IFLAG_HANDLE ; } ret =ioctl (fd,XFS_IOC_GETPARENTS , pi);while (!ret) { -if (pi->pi_flags &XFS_PPTR_OFLAG_ROOT ) { +if (pi->gp_flags &XFS_GETPARENTS_OFLAG_ROOT ) { ret =fn (pi,NULL , arg); goto out_pi; } -for (i =0 ; i < pi->pi_count; i++) { - p =xfs_ppinfo_to_pp (pi, i); +for (i =0 ; i < pi->gp_count; i++) { + p =xfs_getparents_rec (pi, i); ret =fn (pi, p, arg);if (ret) goto out_pi; } -if (pi->pi_flags &XFS_PPTR_OFLAG_DONE ) +if (pi->gp_flags &XFS_GETPARENTS_OFLAG_DONE )break ; ret =ioctl (fd,XFS_IOC_GETPARENTS , pi);
@@ -127,7 +127,7 @@ static int handle_walk_parent_paths(struct walk_ppaths_info *wpi,static int handle_walk_parent_path_ptr ( -struct xfs_pptr_info *pi, +struct xfs_getparents *pi,struct xfs_parent_ptr *p,void *arg) {
@@ -135,7 +135,7 @@ handle_walk_parent_path_ptr(struct walk_ppaths_info *wpi = wpli->wpi;int ret =0 ; -if (pi->pi_flags &XFS_PPTR_OFLAG_ROOT ) +if (pi->gp_flags &XFS_GETPARENTS_OFLAG_ROOT )return wpi->fn (wpi->mntpt, wpi->path, wpi->arg); ret = path_component_change(wpli->pc, p->xpp_name,
libfrog/pptrs.h +1 -1
@@ -8,7 +8,7 @@struct path_list ; -typedef int (*walk_pptr_fn )(struct xfs_pptr_info *pi, +typedef int (*walk_pptr_fn )(struct xfs_getparents *pi,struct xfs_parent_ptr *pptr,void *arg);typedef int (*walk_ppath_fn )(const char *mntpt,struct path_list *path,void *arg);
libxfs/xfs_fs.h +23 -22
91 lines not on this page · raw message
PATCH 2/5 xfs: rename xfs_parent_ptr
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Change the name to xfs_getparents_rec so that the name matches the head structure.
--- io/parent.c | 18 +++++++++--------- libfrog/pptrs.c | 12 ++++++------ libfrog/pptrs.h | 2 +- libxfs/xfs_fs.h | 20 ++++++++++---------- 4 files changed, 26 insertions(+), 26 deletions(-)
io/parent.c +9 -9
@@ -25,7 +25,7 @@ struct pptr_args {static int pptr_print (struct xfs_getparents *pi, -struct xfs_parent_ptr *pptr, +struct xfs_getparents_rec *pptr,void *arg) {struct pptr_args *args = arg;
@@ -36,21 +36,21 @@ pptr_print(return 0 ; } -if (args->filter_ino && pptr->xpp_ino != args->filter_ino) +if (args->filter_ino && pptr->gpr_ino != args->filter_ino)return 0 ; -if (args->filter_name &&strcmp (args->filter_name, pptr->xpp_name)) +if (args->filter_name &&strcmp (args->filter_name, pptr->gpr_name))return 0 ; - namelen =strlen (pptr->xpp_name); + namelen =strlen (pptr->gpr_name);if (args->shortformat) {printf ("%llu/%u/%u/%s\n" , - (unsigned long long )pptr->xpp_ino, - (unsigned int )pptr->xpp_gen, namelen, pptr->xpp_name); + (unsigned long long )pptr->gpr_ino, + (unsigned int )pptr->gpr_gen, namelen, pptr->gpr_name); }else { -printf (_ ("p_ino = %llu\n" ), (unsigned long long )pptr->xpp_ino); -printf (_ ("p_gen = %u\n" ), (unsigned int )pptr->xpp_gen); +printf (_ ("p_ino = %llu\n" ), (unsigned long long )pptr->gpr_ino); +printf (_ ("p_gen = %u\n" ), (unsigned int )pptr->gpr_gen);printf (_ ("p_reclen = %u\n" ), namelen); -printf (_ ("p_name = \"%s\"\n\n" ), pptr->xpp_name); +printf (_ ("p_name = \"%s\"\n\n" ), pptr->gpr_name); }return 0 ; }
libfrog/pptrs.c +6 -6
@@ -37,7 +37,7 @@ handle_walk_parents( void *arg) {struct xfs_getparents *pi; -struct xfs_parent_ptr *p; +struct xfs_getparents_rec *p;unsigned int i;ssize_t ret =-1 ;
@@ -128,7 +128,7 @@ static int handle_walk_parent_paths(struct walk_ppaths_info *wpi,static int handle_walk_parent_path_ptr (struct xfs_getparents *pi, -struct xfs_parent_ptr *p, +struct xfs_getparents_rec *p,void *arg) {struct walk_ppath_level_info *wpli = arg;
@@ -138,13 +138,13 @@ handle_walk_parent_path_ptr(if (pi->gp_flags &XFS_GETPARENTS_OFLAG_ROOT )return wpi->fn (wpi->mntpt, wpi->path, wpi->arg); - ret =path_component_change (wpli->pc, p->xpp_name, -strlen ((char *)p->xpp_name), p->xpp_ino); + ret =path_component_change (wpli->pc, p->gpr_name, +strlen ((char *)p->gpr_name), p->gpr_ino);if (ret)return ret; - wpli->newhandle.ha_fid.fid_ino = p->xpp_ino; - wpli->newhandle.ha_fid.fid_gen = p->xpp_gen; + wpli->newhandle.ha_fid.fid_ino = p->gpr_ino; + wpli->newhandle.ha_fid.fid_gen = p->gpr_gen;path_list_add_parent_component (wpi->path, wpli->pc); ret =handle_walk_parent_paths (wpi, &wpli->newhandle);
libfrog/pptrs.h +1 -1
@@ -9,7 +9,7 @@struct path_list ;typedef int (*walk_pptr_fn )(struct xfs_getparents *pi, -struct xfs_parent_ptr *pptr,void *arg); +struct xfs_getparents_rec *pptr,void *arg);typedef int (*walk_ppath_fn )(const char *mntpt,struct path_list *path,void *arg);
libxfs/xfs_fs.h +10 -10
@@ -768,12 +768,12 @@ struct xfs_scrub_metadata { XFS_GETPARENTS_OFLAG_DONE)/* Get an inode parent pointer through ioctl */ -struct xfs_parent_ptr { -__u64 xpp_ino;/* Inode */ -__u32 xpp_gen;/* Inode generation */ -__u32 xpp_diroffset;/* Directory offset */ -__u64 xpp_rsvd;/* Reserved */ -__u8 xpp_name[];/* File name */ +struct xfs_getparents_rec { +__u64 gpr_ino;/* Inode */ +__u32 gpr_gen;/* Inode generation */ +__u32 gpr_diroffset;/* Directory offset */ +__u64 gpr_rsvd;/* Reserved */ +__u8 gpr_name[];/* File name */ };/* Iterate through an inodes parent pointers */
@@ -811,15 +811,15 @@ static inline size_t xfs_getparents_sizeof(int nr_ptrs) {return sizeof (struct xfs_getparents ) + - (nr_ptrs *sizeof (struct xfs_parent_ptr )); + (nr_ptrs *sizeof (struct xfs_getparents_rec )); } -static inline struct xfs_parent_ptr * +static inline struct xfs_getparents_rec *xfs_getparents_rec (struct xfs_getparents *info,int idx) { -return (struct xfs_parent_ptr *)((char *)info + info->gp_offsets[idx]); +return (struct xfs_getparents_rec *)((char *)info + info->gp_offsets[idx]); }/*
@@ -867,7 +867,7 @@ xfs_getparents_rec(/* XFS_IOC_GETFSMAP ------ hoisted 59 */ #define XFS_IOC_SCRUB_METADATA _IOWR('X', 60, struct xfs_scrub_metadata)#define XFS_IOC_AG_GEOMETRY _IOWR('X', 61, struct xfs_ag_geometry) -#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_parent_ptr) +#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_getparents_rec)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 3/5 xfs: fix GETPARENTS ioctl
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix a few remaining issues with this ioctl:
The ioctl encodes the size of the parent rec, not the parent head.
The parent rec should say that it returns a null terminated filename.
The parent head encodes the buffer size, not the size of the parent record array, but the field name and documentation doesn't make this clear.
The getparents sizeof function is pointless and wrong.
Get rid of the last vestiges of the non-flex-array definitions.
The rec address should take an unsigned argument
Whitespace damage
--- libfrog/pptrs.c | 2 +- libxfs/xfs_fs.h | 30 +++++++++++------------------- 2 files changed, 12 insertions(+), 20 deletions(-)
libfrog/pptrs.c +1 -1
@@ -21,7 +21,7 @@ alloc_pptr_buf( pi =calloc (bufsize,1 );if (!pi)return NULL ; - pi->gp_ptrs_size = bufsize; + pi->gp_bufsize = bufsize;return pi; }
libxfs/xfs_fs.h +11 -19
@@ -752,8 +752,6 @@ struct xfs_scrub_metadata { XFS_SCRUB_OFLAG_NO_REPAIR_NEEDED)#define XFS_SCRUB_FLAGS_ALL (XFS_SCRUB_FLAGS_IN | XFS_SCRUB_FLAGS_OUT) -#define XFS_GETPARENTS_MAXNAMELEN 256 -/* return parents of the handle, not the open fd */ #define XFS_GETPARENTS_IFLAG_HANDLE (1U << 0)
@@ -769,11 +767,11 @@ struct xfs_scrub_metadata {/* Get an inode parent pointer through ioctl */ struct xfs_getparents_rec { -__u64 gpr_ino;/* Inode */ -__u32 gpr_gen;/* Inode generation */ -__u32 gpr_diroffset;/* Directory offset */ -__u64 gpr_rsvd;/* Reserved */ -__u8 gpr_name[];/* File name */ +__u64 gpr_ino;/* Inode number */ +__u32 gpr_gen;/* Inode generation */ +__u32 gpr_diroffset;/* Directory offset */ +__u64 gpr_rsvd;/* Reserved */ +__u8 gpr_name[];/* File name and null terminator */ };/* Iterate through an inodes parent pointers */
@@ -794,8 +792,8 @@ struct xfs_getparents {/* Must be set to zero */ __u32 gp_reserved; -/* size of the trailing buffer in bytes */ -__u32 gp_ptrs_size; +/* Size of the buffer in bytes, including this header */ +__u32 gp_bufsize;/* # of entries filled in (output) */ __u32 gp_count;
@@ -807,19 +805,13 @@ struct xfs_getparents {__u32 gp_offsets[]; }; -static inline size_t -xfs_getparents_sizeof (int nr_ptrs) -{ -return sizeof (struct xfs_getparents ) + - (nr_ptrs *sizeof (struct xfs_getparents_rec )); -} -static inline struct xfs_getparents_rec *xfs_getparents_rec (struct xfs_getparents *info, -int idx) +unsigned int idx) { -return (struct xfs_getparents_rec *)((char *)info + info->gp_offsets[idx]); +return (struct xfs_getparents_rec *)((char *)info + + info->gp_offsets[idx]); }/*
@@ -867,7 +859,7 @@ xfs_getparents_rec(/* XFS_IOC_GETFSMAP ------ hoisted 59 */ #define XFS_IOC_SCRUB_METADATA _IOWR('X', 60, struct xfs_scrub_metadata)#define XFS_IOC_AG_GEOMETRY _IOWR('X', 61, struct xfs_ag_geometry) -#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_getparents_rec) +#define XFS_IOC_GETPARENTS _IOWR('X', 62, struct xfs_getparents)/* * ioctl commands that replace IRIX syssgi()'s
PATCH 4/5 xfs: shorten parent pointer function names
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Shorten the function names and add brief comments to each, outlining what they're supposed to be doing.
--- libxfs/libxfs_api_defs.h | 2 +- libxfs/xfs_parent.c | 18 ++++++++++++------ libxfs/xfs_parent.h | 24 ++++++++++++------------ mkfs/proto.c | 12 ++++++------ 4 files changed, 31 insertions(+), 25 deletions(-)
libxfs/libxfs_api_defs.h +1 -1
@@ -140,7 +140,7 @@#define xfs_log_get_max_trans_res libxfs_log_get_max_trans_res#define xfs_log_sb libxfs_log_sb#define xfs_mode_to_ftype libxfs_mode_to_ftype -#define xfs_parent_defer_add libxfs_parent_defer_add +#define xfs_parent_add libxfs_parent_add#define xfs_parent_finish libxfs_parent_finish#define xfs_parent_start libxfs_parent_start#define xfs_perag_get libxfs_perag_get
libxfs/xfs_parent.c +12 -6
@@ -136,6 +136,10 @@ xfs_parent_irec_from_disk(memset (&irec->p_name[valuelen],0 ,sizeof (irec->p_name) - valuelen); } +/* +* Allocate memory to control a logged parent pointer update as part of a +* dirent operation. +*/ int __xfs_parent_init( struct xfs_mount *mp,
@@ -171,12 +175,13 @@ __xfs_parent_init(return 0 ; } +/* Add a parent pointer to reflect a dirent addition. */ int -xfs_parent_defer_add ( +xfs_parent_add (struct xfs_trans *tp,struct xfs_parent_defer *parent,struct xfs_inode *dp, -struct xfs_name *parent_name, +const struct xfs_name *parent_name,xfs_dir2_dataptr_t diroffset,struct xfs_inode *child) {
@@ -195,8 +200,9 @@ xfs_parent_defer_add(return xfs_attr_defer_add (args); } +/* Remove a parent pointer to reflect a dirent removal. */ int -xfs_parent_defer_remove( +xfs_parent_remove( struct xfs_trans *tp, struct xfs_inode *dp, struct xfs_parent_defer *parent,
@@ -212,14 +218,14 @@ xfs_parent_defer_remove(return xfs_attr_defer_remove (args); } - +/* Replace one parent pointer with another to reflect a rename. */ int -xfs_parent_defer_replace ( +xfs_parent_replace (struct xfs_trans *tp,struct xfs_parent_defer *new_parent,struct xfs_inode *old_dp,xfs_dir2_dataptr_t old_diroffset, -struct xfs_name *parent_name, +const struct xfs_name *parent_name,struct xfs_inode *new_dp,xfs_dir2_dataptr_t new_diroffset,struct xfs_inode *child)
libxfs/xfs_parent.h +12 -12
@@ -49,8 +49,9 @@ struct xfs_parent_defer {* Parent pointer attribute prototypes */ void xfs_init_parent_name_rec (struct xfs_parent_name_rec *rec, -struct xfs_inode *ip, -uint32_t p_diroffset); +struct xfs_inode *ip,uint32_t p_diroffset); +void xfs_init_parent_name_irec (struct xfs_parent_name_irec *irec, +struct xfs_parent_name_rec *rec);int __xfs_parent_init (struct xfs_mount *mp,bool grab_log,struct xfs_parent_defer **parentp);
@@ -78,18 +79,17 @@ xfs_parent_start_locked(return 0 ; } -int xfs_parent_defer_add (struct xfs_trans *tp,struct xfs_parent_defer *parent, -struct xfs_inode *dp,struct xfs_name *parent_name, -xfs_dir2_dataptr_t diroffset,struct xfs_inode *child); -int xfs_parent_defer_replace (struct xfs_trans *tp, +int xfs_parent_add (struct xfs_trans *tp,struct xfs_parent_defer *parent, +struct xfs_inode *dp,const struct xfs_name *parent_name, +xfs_dir2_dataptr_t diroffset,struct xfs_inode *child); +int xfs_parent_replace (struct xfs_trans *tp,struct xfs_parent_defer *new_parent,struct xfs_inode *old_dp, -xfs_dir2_dataptr_t old_diroffset,struct xfs_name *parent_name, -struct xfs_inode *new_ip,xfs_dir2_dataptr_t new_diroffset, +xfs_dir2_dataptr_t old_diroffset, +const struct xfs_name *parent_name,struct xfs_inode *new_ip, +xfs_dir2_dataptr_t new_diroffset,struct xfs_inode *child); +int xfs_parent_remove (struct xfs_trans *tp,struct xfs_inode *dp, +struct xfs_parent_defer *parent,xfs_dir2_dataptr_t diroffset,struct xfs_inode *child); -int xfs_parent_defer_remove (struct xfs_trans *tp,struct xfs_inode *dp, -struct xfs_parent_defer *parent, -xfs_dir2_dataptr_t diroffset, -struct xfs_inode *child);void __xfs_parent_cancel (struct xfs_mount *mp,struct xfs_parent_defer *parent);
mkfs/proto.c +6 -6
@@ -508,8 +508,8 @@ parseproto(newdirent (mp, tp, pip, &xname, ip->i_ino, &offset);libxfs_trans_log_inode (tp, ip, flags);if (parent) { - error = -libxfs_parent_defer_add (tp, parent, pip, - &xname, offset, ip); + error = -libxfs_parent_add (tp, parent, pip, &xname, + offset, ip);if (error)fail (_ ("committing parent pointers failed." ), error);
@@ -601,8 +601,8 @@ parseproto(newdirectory (mp, tp, ip, pip);libxfs_trans_log_inode (tp, ip, flags);if (parent) { - error = -libxfs_parent_defer_add (tp, parent, pip, - &xname, offset, ip); + error = -libxfs_parent_add (tp, parent, pip, &xname, + offset, ip);if (error)fail (_ ("committing parent pointers failed." ), error);
@@ -636,8 +636,8 @@ parseproto( }libxfs_trans_log_inode (tp, ip, flags);if (parent) { - error = -libxfs_parent_defer_add (tp, parent, pip, &xname, - offset, ip); + error = -libxfs_parent_add (tp, parent, pip, &xname, offset, + ip);if (error)fail (_ ("committing parent pointers failed." ), error); }
PATCH 5/5 xfs: rearrange bits of the parent pointer apis for fsck
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Rearrange parts of this thing in preparation for fsck code.
--- libxfs/libxfs_api_defs.h | 1 - libxfs/xfs_da_format.h | 11 +++++++++++ libxfs/xfs_parent.c | 29 ++++++++++++++++++++++++++++- libxfs/xfs_parent.h | 6 ++---- 4 files changed, 41 insertions(+), 6 deletions(-)
libxfs/libxfs_api_defs.h +0 -1
@@ -124,7 +124,6 @@#define xfs_initialize_perag libxfs_initialize_perag#define xfs_initialize_perag_data libxfs_initialize_perag_data#define xfs_init_local_fork libxfs_init_local_fork -#define xfs_init_parent_name_rec libxfs_init_parent_name_rec#define xfs_inobt_maxrecs libxfs_inobt_maxrecs#define xfs_inobt_stage_cursor libxfs_inobt_stage_cursor
libxfs/xfs_da_format.h +11 -0
@@ -159,6 +159,17 @@ struct xfs_da3_intnode {#define XFS_DIR3_FT_MAX 9 +#define XFS_DIR3_FTYPE_STR \ + { XFS_DIR3_FT_UNKNOWN, "unknown" }, \ + { XFS_DIR3_FT_REG_FILE, "file" }, \ + { XFS_DIR3_FT_DIR, "directory" }, \ + { XFS_DIR3_FT_CHRDEV, "char" }, \ + { XFS_DIR3_FT_BLKDEV, "block" }, \ + { XFS_DIR3_FT_FIFO, "fifo" }, \ + { XFS_DIR3_FT_SOCK, "sock" }, \ + { XFS_DIR3_FT_SYMLINK, "symlink" }, \ + { XFS_DIR3_FT_WHT, "whiteout" } +/* * Byte offset in data block and shortform entry. */
libxfs/xfs_parent.c +28 -1
@@ -92,7 +92,7 @@ xfs_parent_valuecheck( }/* Initializes a xfs_parent_name_rec to be stored as an attribute name */ -void +static inline void xfs_init_parent_name_rec( struct xfs_parent_name_rec *rec, struct xfs_inode *ip,
@@ -136,6 +136,33 @@ xfs_parent_irec_from_disk(memset (&irec->p_name[valuelen],0 ,sizeof (irec->p_name) - valuelen); } +/* +* Convert an incore parent_name record to its ondisk format. If @value or +* @valuelen are NULL, they will not be written to. +*/ +void +xfs_parent_irec_to_disk ( +struct xfs_parent_name_rec *rec, +void *value, +int *valuelen, +const struct xfs_parent_name_irec *irec) +{ + rec->p_ino =cpu_to_be64 (irec->p_ino); + rec->p_gen =cpu_to_be32 (irec->p_gen); + rec->p_diroffset =cpu_to_be32 (irec->p_diroffset); + +if (valuelen) { +ASSERT (*valuelen >0 ); +ASSERT (*valuelen >= irec->p_namelen); +ASSERT (*valuelen <MAXNAMELEN ); + + *valuelen = irec->p_namelen; + } + +if (value) +memcpy (value, irec->p_name, irec->p_namelen); +} +/* * Allocate memory to control a logged parent pointer update as part of a * dirent operation.
libxfs/xfs_parent.h +2 -4
@@ -33,6 +33,8 @@ struct xfs_parent_name_irec {void xfs_parent_irec_from_disk (struct xfs_parent_name_irec *irec,const struct xfs_parent_name_rec *rec,const void *value,int valuelen); +void xfs_parent_irec_to_disk (struct xfs_parent_name_rec *rec,void *value, +int *valuelen,const struct xfs_parent_name_irec *irec);/* * Dynamically allocd structure used to wrap the needed data to pass around
@@ -48,10 +50,6 @@ struct xfs_parent_defer {/* * Parent pointer attribute prototypes */ -void xfs_init_parent_name_rec (struct xfs_parent_name_rec *rec, -struct xfs_inode *ip,uint32_t p_diroffset); -void xfs_init_parent_name_irec (struct xfs_parent_name_irec *irec, -struct xfs_parent_name_rec *rec);int __xfs_parent_init (struct xfs_mount *mp,bool grab_log,struct xfs_parent_defer **parentp);
cover xfs_logprint: clean up attri/pptr dumping
In reply to Darrick J. WongHi all,
This series starts by reverting all the recent changes to log_redo.c because there's a lot of duplicated code and overly complicated pointer handling. Next, we address a few missing pieces in the attri log item decoding, and finish the series by adding proper decoding of parent pointers embedded in an xattri log intent item.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-logprint-cleanups
--- libxfs/libxfs_api_defs.h | 1 logprint/log_redo.c | 266 +++++++++++++++++----------------------------- logprint/logprint.h | 5 - 3 files changed, 102 insertions(+), 170 deletions(-)
PATCH 1/4 xfs: revert "xfsprogs: Print pptrs in ATTRI items"
From: Darrick J. Wong <[email protected]>
Parent pointers are stored inside extended attributes, which means that they're a higher level data structure. logprint shouldn't be reporting parent pointers as "ATTRI" items, so let's replace all that with proper helpers that will decode a parent pointer xattri log item.
But first, get rid of all this code that adds a bunch of repetitive logic and unnecessary heap allocations.
--- logprint/log_redo.c | 193 ++++++--------------------------------------------- logprint/logprint.h | 5 + 2 files changed, 26 insertions(+), 172 deletions(-)
logprint/log_redo.c +24 -169
@@ -674,31 +674,6 @@ xfs_attri_copy_log_format(return 1 ; } -/* iovec length must be 32-bit aligned */ -static inline size_t ATTR_NVEC_SIZE (size_t size) -{ -return round_up (size,sizeof (int32_t )); -} - -static int -xfs_attri_copy_name_format ( -char *buf, -uint len, -struct xfs_parent_name_rec *dst_attri_fmt) -{ -uint dst_len =ATTR_NVEC_SIZE (sizeof (struct xfs_parent_name_rec )); - -if (len == dst_len) { -memcpy ((char *)dst_attri_fmt, buf, len); -return 0 ; - } - -fprintf (stderr,_ ("%s: bad size of attri name format: %u; expected %u\n" ), - progname, len, dst_len); - -return 1 ; -} - int xlog_print_trans_attri( char **ptr,
@@ -739,8 +714,7 @@ xlog_print_trans_attri( (*i)++; head = (xlog_op_header_t *)*ptr;xlog_print_op_header (head, *i, ptr); - error =xlog_print_trans_attri_name (ptr,be32_to_cpu (head->oh_len), - src_f->alfi_attr_filter); + error =xlog_print_trans_attri_name (ptr,be32_to_cpu (head->oh_len));if (error) goto error; }
@@ -750,8 +724,7 @@ xlog_print_trans_attri( (*i)++; head = (xlog_op_header_t *)*ptr;xlog_print_op_header (head, *i, ptr); - error =xlog_print_trans_attri_name (ptr,be32_to_cpu (head->oh_len), - src_f->alfi_attr_filter); + error =xlog_print_trans_attri_name (ptr,be32_to_cpu (head->oh_len));if (error) goto error; }
@@ -762,7 +735,7 @@ xlog_print_trans_attri( head = (xlog_op_header_t *)*ptr;xlog_print_op_header (head, *i, ptr); error =xlog_print_trans_attri_value (ptr,be32_to_cpu (head->oh_len), - src_f->alfi_value_len, src_f->alfi_attr_filter); + src_f->alfi_value_len); } error:free (src_f);
@@ -773,45 +746,13 @@ xlog_print_trans_attri(int xlog_print_trans_attri_name (char **ptr, -uint src_len, -uint attr_flags) +uint src_len) { -struct xfs_parent_name_rec *src_f =NULL ; -uint dst_len; +printf (_ ("ATTRI: name len:%u\n" ), src_len); +print_or_dump (*ptr, src_len); -/* -* If this is not a parent pointer, just do a bin dump -*/ -if (!(attr_flags &XFS_ATTR_PARENT )) { -printf (_ ("ATTRI: name len:%u\n" ), src_len); -print_or_dump (*ptr, src_len); - goto out; - } - - dst_len =ATTR_NVEC_SIZE (sizeof (struct xfs_parent_name_rec )); -if (dst_len != src_len) { -fprintf (stderr,_ ("%s: bad size of attri name format: %u; expected %u\n" ), - progname, src_len, dst_len); -return 1 ; - } - -/* -* memmove to ensure 8-byte alignment for the long longs in -* xfs_parent_name_rec structure -*/ -if ((src_f = (struct xfs_parent_name_rec *)malloc (src_len)) ==NULL ) { -fprintf (stderr,_ ("%s: xlog_print_trans_attri_name: malloc failed\n" ), progname); -exit (1 ); - } -memmove ((char *)src_f, *ptr, src_len); - -printf (_ ("ATTRI: #p_ino: %llu p_gen: %u, p_diroffset: %u\n" ), -be64_to_cpu (src_f->p_ino),be32_to_cpu (src_f->p_gen), -be32_to_cpu (src_f->p_diroffset)); - -free (src_f); -out: *ptr += src_len; +return 0 ; }/* xlog_print_trans_attri */
@@ -819,32 +760,15 @@ intxlog_print_trans_attri_value ( char **ptr, uint src_len, - int value_len, - uint attr_flags) + int value_len) {int len =min (value_len, src_len); -char *f =NULL ; -/* -* If this is not a parent pointer, just do a bin dump -*/ -if (!(attr_flags &XFS_ATTR_PARENT )) { -printf (_ ("ATTRI: value len:%u\n" ), value_len); -print_or_dump (*ptr,min (len,MAX_ATTR_VAL_PRINT )); - goto out; - } +printf (_ ("ATTRI: value len:%u\n" ), value_len); +print_or_dump (*ptr, len); -if ((f = (char *)malloc (src_len)) ==NULL ) { -fprintf (stderr,_ ("%s: xlog_print_trans_attri: malloc failed\n" ), progname); -exit (1 ); - } - -memcpy (f, *ptr, value_len); -printf (_ ("ATTRI: value: %.*s\n" ), value_len, f); - -free (f); -out: *ptr += src_len; +return 0 ; }/* xlog_print_trans_attri_value */
@@ -855,9 +779,6 @@ xlog_recover_print_attri(struct xfs_attri_log_format *f, *src_f =NULL ;uint src_len, dst_len; -struct xfs_parent_name_rec *rec, *src_rec =NULL ; -char *value, *src_value =NULL ; -int region =0 ; src_f = (struct xfs_attri_log_format *)item->ri_buf[0 ].i_addr;
@@ -882,93 +803,27 @@ xlog_recover_print_attri(if (f->alfi_name_len >0 ) { region++; - -if (f->alfi_attr_filter &XFS_ATTR_PARENT ) { - src_rec = (struct xfs_parent_name_rec *)item->ri_buf[region].i_addr; - src_len = item->ri_buf[region].i_len; - - dst_len =ATTR_NVEC_SIZE (sizeof (struct xfs_parent_name_rec )); - -if ((rec = ((struct xfs_parent_name_rec *)malloc (dst_len))) ==NULL ) { -fprintf (stderr,_ ("%s: xlog_recover_print_attri: malloc failed\n" ), - progname); -exit (1 ); - } -if (xfs_attri_copy_name_format ((char *)src_rec, src_len, rec)) { - goto out; - } - -printf (_ ("ATTRI: #inode: %llu gen: %u, offset: %u\n" ), -be64_to_cpu (rec->p_ino),be32_to_cpu (rec->p_gen), -be32_to_cpu (rec->p_diroffset)); - -free (rec); - } -else { -printf (_ ("ATTRI: name len:%u\n" ), f->alfi_name_len); -print_or_dump ((char *)item->ri_buf[region].i_addr, - f->alfi_name_len); - } +printf (_ ("ATTRI: name len:%u\n" ), f->alfi_name_len); +print_or_dump ((char *)item->ri_buf[region].i_addr, + f->alfi_name_len); }if (f->alfi_nname_len >0 ) { region++; - -if (f->alfi_attr_filter &XFS_ATTR_PARENT ) { - src_rec = (struct xfs_parent_name_rec *)item->ri_buf[region].i_addr; - src_len = item->ri_buf[region].i_len; - - dst_len =ATTR_NVEC_SIZE (sizeof (struct xfs_parent_name_rec )); - -if ((rec = ((struct xfs_parent_name_rec *)malloc (dst_len))) ==NULL ) { -fprintf (stderr,_ ("%s: xlog_recover_print_attri: malloc failed\n" ), - progname); -exit (1 ); - } -if (xfs_attri_copy_name_format ((char *)src_rec, src_len, rec)) { - goto out; - } - -printf (_ ("ATTRI: new #inode: %llu gen: %u, offset: %u\n" ), -be64_to_cpu (rec->p_ino),be32_to_cpu (rec->p_gen), -be32_to_cpu (rec->p_diroffset)); - -free (rec); - } -else { -printf (_ ("ATTRI: nname len:%u\n" ), f->alfi_nname_len); -print_or_dump ((char *)item->ri_buf[region].i_addr, - f->alfi_nname_len); - } +printf (_ ("ATTRI: nname len:%u\n" ), f->alfi_nname_len); +print_or_dump ((char *)item->ri_buf[region].i_addr, + f->alfi_nname_len); }if (f->alfi_value_len >0 ) { +int len = f->alfi_value_len; + +if (len >MAX_ATTR_VAL_PRINT ) + len =MAX_ATTR_VAL_PRINT ; + region++; - -if (f->alfi_attr_filter &XFS_ATTR_PARENT ) { - src_value = (char *)item->ri_buf[region].i_addr; - -if ((value = ((char *)malloc (f->alfi_value_len))) ==NULL ) { -fprintf (stderr,_ ("%s: xlog_recover_print_attri: malloc failed\n" ), - progname); -exit (1 ); - } - -memcpy ((char *)value, (char *)src_value, f->alfi_value_len); -printf ("ATTRI: value: %.*s\n" , f->alfi_value_len, value); - -free (value); - } -else { -int len = f->alfi_value_len; - -if (len >MAX_ATTR_VAL_PRINT ) - len =MAX_ATTR_VAL_PRINT ; - -printf (_ ("ATTRI: value len:%u\n" ), f->alfi_value_len); -print_or_dump ((char *)item->ri_buf[region].i_addr, - len); - } +printf (_ ("ATTRI: value len:%u\n" ), f->alfi_value_len); +print_or_dump ((char *)item->ri_buf[region].i_addr, len); } out:
logprint/logprint.h +2 -3
@@ -59,9 +59,8 @@ extern void xlog_recover_print_bud(struct xlog_recover_item *item);#define MAX_ATTR_VAL_PRINT 128extern int xlog_print_trans_attri (char **ptr,uint src_len,int *i); -extern int xlog_print_trans_attri_name (char **ptr,uint src_len,uint attr_flags); -extern int xlog_print_trans_attri_value (char **ptr,uint src_len,int value_len, -uint attr_flags); +extern int xlog_print_trans_attri_name (char **ptr,uint src_len); +extern int xlog_print_trans_attri_value (char **ptr,uint src_len,int value_len);extern void xlog_recover_print_attri (struct xlog_recover_item *item);extern int xlog_print_trans_attrd (char **ptr,uint len);extern void xlog_recover_print_attrd (struct xlog_recover_item *item);
PATCH 2/4 xfs_logprint: print missing attri header fields
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Not sure why logprint doesn't print the op flags, inode, or attr filter fields. Make it do that.
--- logprint/log_redo.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
logprint/log_redo.c +18 -5
@@ -705,9 +705,15 @@ xlog_print_trans_attri(memmove ((char *)src_f, *ptr, src_len); *ptr += src_len; -printf (_ ("ATTRI: #regs: %d name_len: %d, nname_len: %d value_len: %d id: 0x%llx\n" ), - src_f->alfi_size, src_f->alfi_name_len, src_f->alfi_nname_len, - src_f->alfi_value_len, (unsigned long long )src_f->alfi_id); +printf (_ ("ATTRI: #regs: %d f: 0x%x, ino: 0x%llx, attr_filter: 0x%x, name_len: %d, newname_len: %d, value_len: %d id: 0x%llx\n" ), + src_f->alfi_size, + src_f->alfi_op_flags, + (unsigned long long )src_f->alfi_ino, + src_f->alfi_attr_filter, + src_f->alfi_name_len, + src_f->alfi_nname_len, + src_f->alfi_value_len, + (unsigned long long )src_f->alfi_id);if (src_f->alfi_name_len >0 ) {printf (_ ("\n" ));
@@ -798,8 +804,15 @@ xlog_recover_print_attri(if (xfs_attri_copy_log_format ((char *)src_f, src_len, f)) goto out; -printf (_ ("ATTRI: #regs: %d name_len: %d, nname_len:%d, value_len: %d id: 0x%llx\n" ), - f->alfi_size, f->alfi_name_len, f->alfi_nname_len, f->alfi_value_len, (unsigned long long )f->alfi_id); +printf (_ ("ATTRI: #regs: %d f: 0x%x, ino: 0x%llx, attr_filter: 0x%x, name_len: %d, newname_len:%d, value_len: %d id: 0x%llx\n" ), + f->alfi_size, + f->alfi_op_flags, + (unsigned long long )f->alfi_ino, + f->alfi_attr_filter, + f->alfi_name_len, + f->alfi_nname_len, + f->alfi_value_len, + (unsigned long long )f->alfi_id);if (f->alfi_name_len >0 ) { region++;
PATCH 3/4 xfs: make logprint note attr names and newnames consistently
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix logprint to print "ATTRI: name" and "ATTRI: newname" consistently.
--- logprint/log_redo.c | 15 +++++++++------ logprint/logprint.h | 2 +- 2 files changed, 10 insertions(+), 7 deletions(-)
logprint/log_redo.c +9 -6
@@ -720,7 +720,8 @@ xlog_print_trans_attri( (*i)++; head = (xlog_op_header_t *)*ptr;xlog_print_op_header (head, *i, ptr); - error =xlog_print_trans_attri_name (ptr,be32_to_cpu (head->oh_len)); + error =xlog_print_trans_attri_name (ptr, +be32_to_cpu (head->oh_len),"name" );if (error) goto error; }
@@ -730,7 +731,8 @@ xlog_print_trans_attri( (*i)++; head = (xlog_op_header_t *)*ptr;xlog_print_op_header (head, *i, ptr); - error =xlog_print_trans_attri_name (ptr,be32_to_cpu (head->oh_len)); + error =xlog_print_trans_attri_name (ptr, +be32_to_cpu (head->oh_len),"newname" );if (error) goto error; }
@@ -752,15 +754,16 @@ xlog_print_trans_attri(int xlog_print_trans_attri_name (char **ptr, -uint src_len) +uint src_len, +const char *tag) { -printf (_ ("ATTRI: name len:%u\n" ), src_len); +printf (_ ("ATTRI: %s len:%u\n" ), tag, src_len);print_or_dump (*ptr, src_len); *ptr += src_len;return 0 ; -}/* xlog_print_trans_attri */ +}int xlog_print_trans_attri_value(
@@ -823,7 +826,7 @@ xlog_recover_print_attri(if (f->alfi_nname_len >0 ) { region++; -printf (_ ("ATTRI: nname len:%u\n" ), f->alfi_nname_len); +printf (_ ("ATTRI: newname len:%u\n" ), f->alfi_nname_len);print_or_dump ((char *)item->ri_buf[region].i_addr, f->alfi_nname_len); }
logprint/logprint.h +1 -1
@@ -59,7 +59,7 @@ extern void xlog_recover_print_bud(struct xlog_recover_item *item);#define MAX_ATTR_VAL_PRINT 128extern int xlog_print_trans_attri (char **ptr,uint src_len,int *i); -extern int xlog_print_trans_attri_name (char **ptr,uint src_len); +extern int xlog_print_trans_attri_name (char **ptr,uint src_len,const char *tag);extern int xlog_print_trans_attri_value (char **ptr,uint src_len,int value_len);extern void xlog_recover_print_attri (struct xlog_recover_item *item);extern int xlog_print_trans_attrd (char **ptr,uint len);
PATCH 4/4 xfs_logprint: decode parent pointers fully
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Decode logged parent pointers fully when dumping log contents. Between the existing ATTRI: printouts and the new ones introduced here, we can figure out what was stored in each log iovec, as well as the higher level parent pointer that was logged.
--- libxfs/libxfs_api_defs.h | 1 + logprint/log_redo.c | 63 +++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 63 insertions(+), 1 deletion(-)
libxfs/libxfs_api_defs.h +1 -0
@@ -141,6 +141,7 @@#define xfs_mode_to_ftype libxfs_mode_to_ftype#define xfs_parent_add libxfs_parent_add#define xfs_parent_finish libxfs_parent_finish +#define xfs_parent_irec_from_disk libxfs_parent_irec_from_disk#define xfs_parent_start libxfs_parent_start#define xfs_perag_get libxfs_perag_get#define xfs_perag_put libxfs_perag_put
logprint/log_redo.c +62 -1
122 lines not on this page · raw message
cover fstests: adjust tests for xfs parent pointers
In reply to Darrick J. WongHi all,
Adjust fstests as necessary to test the new xfs parent pointers feature. At some point this section needs to grow some specific functionality tests for repair and dumping.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs
---
common/parent | 209 +++++++
common/populate | 38 +
common/rc | 7
common/xfs | 12
doc/group-names.txt | 1
src/popdir.pl | 11
tests/generic/050 | 10
tests/generic/050.cfg | 1
tests/generic/050.out.xfsquotaparent | 23 +
tests/xfs/018 | 7
tests/xfs/021 | 15 -
tests/xfs/021.cfg | 1
tests/xfs/021.out.default | 0
tests/xfs/021.out.parent | 64 ++
tests/xfs/122.out | 3
tests/xfs/191 | 7
tests/xfs/206 | 3
tests/xfs/288 | 7
tests/xfs/306 | 9
tests/xfs/851 | 116 ++++
tests/xfs/851.out | 69 ++
tests/xfs/852 | 69 ++
tests/xfs/852.out | 1002 ++++++++++++++++++++++++++++++++++
tests/xfs/853 | 85 +++
tests/xfs/853.out | 14
25 files changed, 1772 insertions(+), 11 deletions(-)
create mode 100644 common/parent
create mode 100644 tests/generic/050.out.xfsquotaparent
create mode 100644 tests/xfs/021.cfg
rename tests/xfs/{021.out => 021.out.default} (100%)
create mode 100644 tests/xfs/021.out.parent
create mode 100755 tests/xfs/851
create mode 100644 tests/xfs/851.out
create mode 100755 tests/xfs/852
create mode 100644 tests/xfs/852.out
create mode 100755 tests/xfs/853
create mode 100644 tests/xfs/853.outPATCH 1/14 xfs/122: update for parent pointers
From: Darrick J. Wong <[email protected]>
Update test for parent pointers.
--- tests/xfs/122.out | 3 +++ tests/xfs/206 | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-)
tests/xfs/122.out +3 -0
@@ -109,7 +109,10 @@ sizeof(struct xfs_legacy_timestamp) = 8 sizeof(struct xfs_log_dinode) = 176 sizeof(struct xfs_log_legacy_timestamp) = 8 sizeof(struct xfs_map_extent) = 32 +sizeof(struct xfs_parent_name_rec) = 16 +sizeof(struct xfs_parent_ptr) = 280 sizeof(struct xfs_phys_extent) = 16 +sizeof(struct xfs_pptr_info) = 104 sizeof(struct xfs_refcount_key) = 4 sizeof(struct xfs_refcount_rec) = 12 sizeof(struct xfs_rmap_key) = 20
tests/xfs/206 +2 -1
@@ -66,7 +66,8 @@ mkfs_filter() -e "/.*crc=/d" \ -e "/^Default configuration/d" \ -e "/metadir=.*/d" \ - -e '/rgcount=/d' + -e '/rgcount=/d' \ + -e '/parent=/d' } # mkfs slightly smaller than that, small log for speed.
PATCH 2/14 populate: create hardlinks for parent pointers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Create some hardlinked files so that we can exercise parent pointers.
--- common/populate | 38 ++++++++++++++++++++++++++++++++++++++ src/popdir.pl | 11 +++++++++++ 2 files changed, 49 insertions(+)
common/populate +38 -0
@@ -376,6 +376,7 @@ _scratch_xfs_populate() { is_rt="$(_xfs_get_rtextents "$SCRATCH_MNT")" is_rmapbt="$(_xfs_has_feature "$SCRATCH_MNT" rmapbt -v)" is_reflink="$(_xfs_has_feature "$SCRATCH_MNT" reflink -v)" + is_pptr="$(_xfs_has_feature "$SCRATCH_MNT" parent -v)" # Reverse-mapping btree if [ $is_rmapbt -gt 0 ]; then
@@ -412,6 +413,43 @@ _scratch_xfs_populate() { cp --reflink=always "${SCRATCH_MNT}/REFCOUNTBT" "${SCRATCH_MNT}/REFCOUNTBT2" fi + # Parent pointers + if [ $is_pptr -gt 0 ]; then + echo "+ parent pointers" + + # Create a couple of parent pointers + __populate_create_dir "${SCRATCH_MNT}/PPTRS" 1 '' --hardlink --format "two_%d" + + # Create one xattr leaf block of parent pointers + nr="$((blksz * 2 / 16))" + __populate_create_dir "${SCRATCH_MNT}/PPTRS" ${nr} '' --hardlink --format "many%04d" + + # Create multiple xattr leaf blocks of large parent pointers + nr="$((blksz * 16 / 16))" + __populate_create_dir "${SCRATCH_MNT}/PPTRS" ${nr} '' --hardlink --format "y%0254d" + + # Create multiple paths to a file + local moof="${SCRATCH_MNT}/PPTRS/moofile" + touch "${moof}" + for ((i = 0; i < 4; i++)); do + mkdir -p "${SCRATCH_MNT}/PPTRS/SUB${i}" + ln "${moof}" "${SCRATCH_MNT}/PPTRS/SUB${i}/moofile" + done + + # Create parent pointers of various lengths + touch "${SCRATCH_MNT}/PPTRS/vlength" + local len_len + local tst + local fname + ln "${SCRATCH_MNT}/PPTRS/vlength" "${SCRATCH_MNT}/PPTRS/b" + for len in 32 64 96 128 160 192 224 250 255; do + len_len="${#len}" + tst="$(perl -e "print \"b\" x (${len} - (${len_len} + 1))")" + fname="v${tst}${len}" + ln "${SCRATCH_MNT}/PPTRS/vlength" "${SCRATCH_MNT}/PPTRS/${fname}" + done + fi + # Copy some real files (xfs tests, I guess...) echo "+ real files" test $fill -ne 0 && __populate_fill_fs "${SCRATCH_MNT}" 5
src/popdir.pl +11 -0
@@ -17,6 +17,7 @@ GetOptions("start=i" => \$start, "dir=s" => \$dir, "remove!" => \$remove, "help!" => \$help, + "hardlink!" => \$hardlink, "verbose!" => \$verbose);
@@ -36,6 +37,7 @@ Options: --format=str printf formatting string for file name ("%08d") --verbose verbose output --help this help screen + --hardlink hardlink subsequent files to the first one created EOF exit(1) unless defined $help; # otherwise...
@@ -51,12 +53,21 @@ $file_mult = 20 if (!defined $file_mult); $format = "%08d" if (!defined $format); $incr = 1 if (!defined $incr); +if ($hardlink) { + $file_mult = 0; + $link_fname = sprintf($format, $start); +} + for ($i = $start; $i <= $end; $i += $incr) { $fname = sprintf($format, $i); if ($remove) { $verbose && print "rm $fname\n"; unlink($fname) or rmdir($fname) or die("unlink $fname"); + } elsif ($hardlink && $i > $start) { + # hardlink the first file + $verbose && print "ln $link_fname $fname\n"; + link $link_fname, $fname; } elsif ($file_mult == 0 or ($i % $file_mult) == 0) { # create a file $verbose && print "touch $fname\n";
PATCH 3/14 xfs/021: adapt golden output files for parent pointers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Parent pointers change the xattr structure dramatically, so fix this test to handle them.
---
common/rc | 4 +++
tests/xfs/021 | 15 +++++++++--
tests/xfs/021.cfg | 1 +
tests/xfs/021.out.default | 0
tests/xfs/021.out.parent | 64 +++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 82 insertions(+), 2 deletions(-)
create mode 100644 tests/xfs/021.cfg
rename tests/xfs/{021.out => 021.out.default} (100%)
create mode 100644 tests/xfs/021.out.parentcommon/rc +4 -0
@@ -3312,6 +3312,8 @@ _get_os_name() _link_out_file_named() { + test -n "$seqfull" || _fail "need to set seqfull" + local features=$2 local suffix=$(FEATURES="$features" perl -e ' my %feathash;
@@ -3347,6 +3349,8 @@ _link_out_file() { local features + test -n "$seqfull" || _fail "need to set seqfull" + if [ $# -eq 0 ]; then features="$(_get_os_name),$FSTYP" if [ -n "$MOUNT_OPTIONS" ]; then
tests/xfs/021 +13 -2
@@ -67,6 +67,13 @@ _scratch_mkfs_xfs >/dev/null \ echo "*** mount FS" _scratch_mount +seqfull=$0 +if _xfs_has_feature $SCRATCH_MNT parent; then + _link_out_file "parent" +else + _link_out_file "" +fi + testfile=$SCRATCH_MNT/testfile echo "*** make test file 1"
@@ -108,7 +115,10 @@ _scratch_unmount >>$seqres.full 2>&1 \ echo "*** dump attributes (1)" _scratch_xfs_db -r -c "inode $inum_1" -c "print a.sfattr" | \ - sed -e '/secure = /d' | sed -e '/parent = /d' + perl -ne ' +/\.secure/ && next; +/\.parent/ && next; + print unless /^\d+:\[.*/;' echo "*** dump attributes (2)"
@@ -124,10 +134,11 @@ s/info.hdr/info/; /hdr.info.uuid/ && next; /hdr.info.lsn/ && next; /hdr.info.owner/ && next; +/\.parent/ && next; s/^(hdr.info.magic =) 0x3bee/\1 0xfbee/; s/^(hdr.firstused =) (\d+)/\1 FIRSTUSED/; s/^(hdr.freemap\[0-2] = \[base,size]).*/\1 [FREEMAP..]/; -s/^(entries\[0-2] = \[hashval,nameidx,incomplete,root,local]).*/\1 [ENTRIES..]/; +s/^(entries\[0-[23]] = \[hashval,nameidx,incomplete,root,local]).*/\1 [ENTRIES..]/; print unless /^\d+:\[.*/;' echo "*** done"
tests/xfs/021.cfg new file +1 -0
@@ -0,0 +1 @@ +parent: parent
tests/xfs/021.out.default renamed from tests/xfs/021.out +0 -0
tests/xfs/021.out.parent new file +64 -0
64 lines not on this page · raw message
PATCH 4/14 generic/050: adapt for parent pointers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Fix this test when quotas and parent pointers are enabled.
--- tests/generic/050 | 10 ++++++++++ tests/generic/050.cfg | 1 + tests/generic/050.out.xfsquotaparent | 23 +++++++++++++++++++++++ 3 files changed, 34 insertions(+) create mode 100644 tests/generic/050.out.xfsquotaparent
tests/generic/050 +10 -0
@@ -36,6 +36,16 @@ elif [ "$FSTYP" = "xfs" ] && echo "$MOUNT_OPTIONS" | grep -q quota ; then # Mounting with quota on XFS requires a writable fs, which means # we expect to fail the ro blockdev test with with EPERM. features="xfsquota" + + if _xfs_has_feature $SCRATCH_DEV parent && \ + ! _xfs_has_feature $SCRATCH_DEV realtime; then + # If we have quotas and parent pointers enabled, the primary + # superblock will be written out with the quota flags set when + # the logged xattrs log_incompat feature is set. Hence the + # norecovery mount won't fail due to quota rejecting the + # mismatch between the mount qflags and the ondisk ones. + features="xfsquotaparent" + fi fi _link_out_file "$features"
tests/generic/050.cfg +1 -0
@@ -1,2 +1,3 @@ nojournal: nojournal xfsquota: xfsquota +xfsquotaparent: xfsquotaparent
tests/generic/050.out.xfsquotaparent new file +23 -0
@@ -0,0 +1,23 @@ +QA output created by 050 +setting device read-only +mounting read-only block device: +mount: SCRATCH_MNT: permission denied +unmounting read-only filesystem +umount: SCRATCH_DEV: not mounted +setting device read-write +mounting read-write block device: +touch files +going down: +unmounting shutdown filesystem: +setting device read-only +mounting filesystem that needs recovery on a read-only device: +mount: device write-protected, mounting read-only +mount: cannot mount device read-only +unmounting read-only filesystem +umount: SCRATCH_DEV: not mounted +mounting filesystem with -o norecovery on a read-only device: +mount: device write-protected, mounting read-only +unmounting read-only filesystem +setting device read-write +mounting filesystem that needs recovery with -o ro: +*** done
PATCH 5/14 xfs/018: disable parent pointers for this test
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
This test depends heavily on the xattr formats created for new files. Parent pointers break those assumptions, so force parent pointers off.
--- tests/xfs/018 | 7 ++++++- tests/xfs/191 | 7 ++++++- tests/xfs/288 | 7 ++++++- 3 files changed, 18 insertions(+), 3 deletions(-)
tests/xfs/018 +6 -1
@@ -100,7 +100,12 @@ attr32l="X$attr32k" attr64k="$attr32k$attr32k" echo "*** mkfs" -_scratch_mkfs >/dev/null + +# Parent pointers change the xattr formats sufficiently to break this test. +# Disable parent pointers if mkfs supports it. +mkfs_args=() +$MKFS_XFS_PROG 2>&1 | grep -q parent=0 && mkfs_args+=(-n parent=0) +_scratch_mkfs "${mkfs_args[@]}" >/dev/null blk_sz=$(_scratch_xfs_get_sb_field blocksize) err_inj_attr_sz=$(( blk_sz / 3 - 50 ))
tests/xfs/191 +6 -1
@@ -33,7 +33,12 @@ _fixed_by_kernel_commit 7be3bd8856fb "xfs: empty xattr leaf header blocks are no _fixed_by_kernel_commit e87021a2bc10 "xfs: use larger in-core attr firstused field and detect overflow" _fixed_by_git_commit xfsprogs f50d3462c654 "xfs_repair: ignore empty xattr leaf blocks" -_scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs +# Parent pointers change the xattr formats sufficiently to break this test. +# Disable parent pointers if mkfs supports it. +mkfs_args=() +$MKFS_XFS_PROG 2>&1 | grep -q parent=0 && mkfs_args+=(-n parent=0) + +_scratch_mkfs_xfs "${mkfs_args[@]}" | _filter_mkfs >$seqres.full 2>$tmp.mkfs cat $tmp.mkfs >> $seqres.full source $tmp.mkfs _scratch_mount
tests/xfs/288 +6 -1
@@ -19,8 +19,13 @@ _supported_fs xfs _require_scratch _require_attrs +# Parent pointers change the xattr formats sufficiently to break this test. +# Disable parent pointers if mkfs supports it. +mkfs_args=() +$MKFS_XFS_PROG 2>&1 | grep -q parent=0 && mkfs_args+=(-n parent=0) + # get block size ($dbsize) from the mkfs output -_scratch_mkfs_xfs 2>/dev/null | _filter_mkfs 2>$tmp.mkfs >/dev/null +_scratch_mkfs_xfs "${mkfs_args[@]}" 2>/dev/null | _filter_mkfs 2>$tmp.mkfs >/dev/null . $tmp.mkfs _scratch_mount
PATCH 6/14 xfs/306: fix formatting failures with parent pointers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
The parent pointers feature isn't supported on tiny 20MB filesystems because the larger directory transactions result in larger minimum log sizes, particularly with nrext64 enabled:
** mkfs failed with extra mkfs options added to " -m rmapbt=0, -i nrext64=1, -n parent=1," by test 306 ** ** attempting to mkfs using only test 306 options: -d size=20m -n size=64k **
max log size 5108 smaller than min log size 5310, filesystem is too small
We don't support 20M filesystems anymore, so bump the filesystem size up to 100M and skip this test if we can't actually format the filesystem. Convert the open-coded punch-alternating logic into a call to that program to reduce execve overhead, which more than makes up having to write 5x as much data to fragment the free space.
--- tests/xfs/306 | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
tests/xfs/306 +4 -5
@@ -23,6 +23,7 @@ _supported_fs xfs _require_scratch_nocheck # check complains about single AG fs _require_xfs_io_command "fpunch" _require_command $UUIDGEN_PROG uuidgen +_require_test_program "punch-alternating" # Disable the scratch rt device to avoid test failures relating to the rt # bitmap consuming all the free space in our small data device.
@@ -30,7 +31,8 @@ unset SCRATCH_RTDEV # Create a small fs with a large directory block size. We want to fill up the fs # quickly and then create multi-fsb dirblocks over fragmented free space. -_scratch_mkfs_xfs -d size=20m -n size=64k >> $seqres.full 2>&1 +_scratch_mkfs_xfs -d size=100m -n size=64k >> $seqres.full 2>&1 || \ + _notrun 'could not format tiny scratch fs' _scratch_mount # Fill a source directory with many largish-named files. 1k uuid-named entries
@@ -49,10 +51,7 @@ done $XFS_IO_PROG -xc "resblks 16" $SCRATCH_MNT >> $seqres.full 2>&1 dd if=/dev/zero of=$SCRATCH_MNT/file bs=4k >> $seqres.full 2>&1 $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/file >> $seqres.full 2>&1 -size=`_get_filesize $SCRATCH_MNT/file` -for i in $(seq 0 8192 $size); do - $XFS_IO_PROG -c "fpunch $i 4k" $SCRATCH_MNT/file >> $seqres.full 2>&1 -done +$here/src/punch-alternating $SCRATCH_MNT/file # Replicate the src dir several times into fragmented free space. After one or # two dirs, we should have nothing but non-contiguous directory blocks.
PATCH 7/14 common: add helpers for parent pointer tests
In reply to Darrick J. WongFrom: Allison Henderson <[email protected]>
Add helper functions in common/parent to parse and verify parent pointers. Also add functions to check that mkfs, kernel, and xfs_io support parent pointers.
--- common/parent | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ common/rc | 3 + common/xfs | 12 +++ 3 files changed, 213 insertions(+) create mode 100644 common/parent
common/parent new file +198 -0
@@ -0,0 +1,198 @@ +# +# Parent pointer common functions +# + +# +# parse_parent_pointer parents parent_inode parent_pointer_name +# +# Given a list of parent pointers, find the record that matches +# the given inode and filename +# +# inputs: +# parents : A list of parent pointers in the format of: +# inode/generation/name_length/name +# parent_inode : The parent inode to search for +# parent_name : The parent name to search for +# +# outputs: +# PPINO : Parent pointer inode +# PPGEN : Parent pointer generation +# PPNAME : Parent pointer name +# PPNAME_LEN : Parent pointer name length +# +_parse_parent_pointer() +{ + local parents=$1 + local pino=$2 + local parent_pointer_name=$3 + + local found=0 + + # Find the entry that has the same inode as the parent + # and parse out the entry info + while IFS=\/ read PPINO PPGEN PPNAME_LEN PPNAME; do + if [ "$PPINO" != "$pino" ]; then + continue + fi + + if [ "$PPNAME" != "$parent_pointer_name" ]; then + continue + fi + + found=1 + break + done <<< $(echo "$parents") + + # Check to see if we found anything + # We do not fail the test because we also use this + # routine to verify when parent pointers should + # be removed or updated (ie a rename or a move + # operation changes your parent pointer) + if [ $found -eq "0" ]; then + return 1 + fi + + # Verify the parent pointer name length is correct + if [ "$PPNAME_LEN" -ne "${#parent_pointer_name}" ] + then + echo "*** Bad parent pointer:"\ + "name:$PPNAME, namelen:$PPNAME_LEN" + fi + + #return sucess + return 0 +} + +# +# _verify_parent parent_path parent_pointer_name child_path +# +# Verify that the given child path lists the given parent as a parent pointer +# and that the parent pointer name matches the given name +# +# Examples: +# +# #simple example +# mkdir testfolder1 +# touch testfolder1/file1 +# verify_parent testfolder1 file1 testfolder1/file1 +# +# # In this above example, we want to verify that "testfolder1" +# # appears as a parent pointer of "testfolder1/file1". Additionally +# # we verify that the name record of the parent pointer is "file1" +# +# +# #hardlink example +# mkdir testfolder1 +# mkdir testfolder2 +# touch testfolder1/file1 +# ln testfolder1/file1 testfolder2/file1_ln +# verify_parent testfolder2 file1_ln testfolder1/file1 +# +# # In this above example, we want to verify that "testfolder2" +# # appears as a parent pointer of "testfolder1/file1". Additionally +# # we verify that the name record of the parent pointer is "file1_ln" +# +_verify_parent() +{ + local parent_path=$1 + local parent_pointer_name=$2 + local child_path=$3 + + local parent_ppath="$parent_path/$parent_pointer_name" + + # Verify parent exists + if [ ! -d $SCRATCH_MNT/$parent_path ]; then + _fail "$SCRATCH_MNT/$parent_path not found" + else + echo "*** $parent_path OK" + fi + + # Verify child exists + if [ ! -f $SCRATCH_MNT/$child_path ]; then + _fail "$SCRATCH_MNT/$child_path not found" + else + echo "*** $child_path OK" + fi + + # Verify the parent pointer name exists as a child of the parent + if [ ! -f $SCRATCH_MNT/$parent_ppath ]; then + _fail "$SCRATCH_MNT/$parent_ppath not found" + else + echo "*** $parent_ppath OK" + fi + + # Get the inodes of both parent and child + pino="$(stat -c '%i' $SCRATCH_MNT/$parent_path)" + cino="$(stat -c '%i' $SCRATCH_MNT/$child_path)" + + # Get all the parent pointers of the child + parents=($($XFS_IO_PROG -x -c \ + "parent -f -i $pino -n $parent_pointer_name" $SCRATCH_MNT/$child_path)) + if [[ $? != 0 ]]; then + _fail "No parent pointers found for $child_path" + fi + + # Parse parent pointer output. + # This sets PPINO PPGEN PPNAME PPNAME_LEN + _parse_parent_pointer $parents $pino $parent_pointer_name + + # If we didnt find one, bail out + if [ $? -ne 0 ]; then + _fail "No parent pointer record found for $parent_path"\ + "in $child_path" + fi + + # Verify the inode generated by the parent pointer name is + # the same as the child inode + pppino="$(stat -c '%i' $SCRATCH_MNT/$parent_ppath)" + if [ $cino -ne $pppino ] + then + _fail "Bad parent pointer name value for $child_path."\ + "$SCRATCH_MNT/$parent_ppath belongs to inode $PPPINO,"\ + "but should be $cino" + fi + + echo "*** Verified parent pointer:"\ + "name:$PPNAME, namelen:$PPNAME_LEN" + echo "*** Parent pointer OK for child $child_path" +} + +# +# _verify_parent parent_pointer_name pino child_path +# +# Verify that the given child path contains no parent pointer entry +# for the given inode and file name +# +_verify_no_parent() +{ + local parent_pname=$1 + local pino=$2 + local child_path=$3 + + # Verify child exists + if [ ! -f $SCRATCH_MNT/$child_path ]; then + _fail "$SCRATCH_MNT/$child_path not found" + else + echo "*** $child_path OK" + fi + + # Get all the parent pointers of the child + local parents=($($XFS_IO_PROG -x -c \ + "parent -f -i $pino -n $parent_pname" $SCRATCH_MNT/$child_path)) + if [[ $? != 0 ]]; then + return 0 + fi + + # Parse parent pointer output. + # This sets PPINO PPGEN PPNAME PPNAME_LEN + _parse_parent_pointer $parents $pino $parent_pname + + # If we didnt find one, return sucess + if [ $? -ne 0 ]; then + return 0 + fi + + _fail "Parent pointer entry found where none should:"\ + "inode:$PPINO, gen:$PPGEN," + "name:$PPNAME, namelen:$PPNAME_LEN" +}
common/rc +3 -0
@@ -2570,6 +2570,9 @@ _require_xfs_io_command() echo $testio | grep -q "invalid option" && \ _notrun "xfs_io $command support is missing" ;; + "parent") + testio=`$XFS_IO_PROG -x -c "parent" $TEST_DIR 2>&1` + ;; "pwrite") # -N (RWF_NOWAIT) only works with direct vectored I/O writes local pwrite_opts=" "
common/xfs +12 -0
@@ -2091,3 +2091,15 @@ _scratch_find_rt_metadir_entry() { grep "${sfkey}.inumber" | awk '{print $1}' return 0 } + +# this test requires the xfs parent pointers feature +# +_require_xfs_parent() +{ + _scratch_mkfs_xfs_supported -n parent > /dev/null 2>&1 \ + || _notrun "mkfs.xfs does not support parent pointers" + _scratch_mkfs_xfs -n parent > /dev/null 2>&1 + _try_scratch_mount >/dev/null 2>&1 \ + || _notrun "kernel does not support parent pointers" + _scratch_unmount +}
PATCH 8/14 xfs: add parent pointer test
In reply to Darrick J. WongFrom: Allison Henderson <[email protected]>
Add a test to verify basic parent pointers operations (create, move, link, unlink, rename, overwrite).
--- doc/group-names.txt | 1 + tests/xfs/851 | 101 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/851.out | 59 ++++++++++++++++++++++++++++++ 3 files changed, 161 insertions(+) create mode 100755 tests/xfs/851 create mode 100644 tests/xfs/851.out
doc/group-names.txt +1 -0
@@ -83,6 +83,7 @@ nfs4_acl NFSv4 access control lists nonsamefs overlayfs layers on different filesystems online_repair online repair functionality tests other dumping ground, do not add more tests to this group +parent Parent pointer tests pattern specific IO pattern tests perms access control and permission checking pipe pipe functionality
tests/xfs/851 new file +101 -0
101 lines not on this page · raw message
tests/xfs/851.out new file +59 -0
59 lines not on this page · raw message
PATCH 9/14 xfs: add multi link parent pointer test
In reply to Darrick J. WongFrom: Allison Henderson <[email protected]>
Add a test to verify parent pointers while multiple links to a file are created and removed.
--- tests/xfs/852 | 69 ++++ tests/xfs/852.out | 1002 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1071 insertions(+) create mode 100755 tests/xfs/852 create mode 100644 tests/xfs/852.out
tests/xfs/852 new file +69 -0
@@ -0,0 +1,69 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022, Oracle and/or its affiliates. All Rights Reserved. +# +# FS QA Test 852 +# +# multi link parent pointer test +# +. ./common/preamble +_begin_fstest auto quick parent + +# get standard environment, filters and checks +. ./common/parent + +# Modify as appropriate +_supported_fs xfs +_require_scratch +_require_xfs_sysfs debug/larp +_require_xfs_parent +_require_xfs_io_command "parent" + +# real QA test starts here + +# Create a directory tree using a protofile and +# make sure all inodes created have parent pointers + +protofile=$tmp.proto + +cat >$protofile <<EOF +DUMMY1 +0 0 +: root directory +d--777 3 1 +: a directory +testfolder1 d--755 3 1 +file1 ---755 3 1 /dev/null +: done +$ +EOF + +_scratch_mkfs -f -n parent=1 -p $protofile >>$seqresres.full 2>&1 \ + || _fail "mkfs failed" +_check_scratch_fs + +_scratch_mount >>$seqres.full 2>&1 \ + || _fail "mount failed" + +testfolder1="testfolder1" +file1="file1" +file1_ln="file1_link" + +echo "" +# Multi link parent pointer test +NLINKS=100 +for (( j=0; j<$NLINKS; j++ )); do + ln $SCRATCH_MNT/$testfolder1/$file1 $SCRATCH_MNT/$testfolder1/$file1_ln.$j + _verify_parent "$testfolder1" "$file1_ln.$j" "$testfolder1/$file1" + _verify_parent "$testfolder1" "$file1" "$testfolder1/$file1_ln.$j" +done +# Multi unlink parent pointer test +for (( j=$NLINKS-1; j<=0; j-- )); do + ino="$(stat -c '%i' $SCRATCH_MNT/$testfolder1/$file1_ln.$j)" + rm $SCRATCH_MNT/$testfolder1/$file1_ln.$j + _verify_no_parent "$file1_ln.$j" "$ino" "$testfolder1/$file1" +done + +# success, all done +status=0 +exit
tests/xfs/852.out new file +1002 -0
1,002 lines not on this page · raw message
PATCH 10/14 xfs: add parent pointer inject test
In reply to Darrick J. WongFrom: Allison Henderson <[email protected]>
Add a test to verify parent pointers after an error injection and log replay.
--- tests/xfs/853 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/853.out | 14 +++++++++ 2 files changed, 99 insertions(+) create mode 100755 tests/xfs/853 create mode 100644 tests/xfs/853.out
tests/xfs/853 new file +85 -0
@@ -0,0 +1,85 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022, Oracle and/or its affiliates. All Rights Reserved. +# +# FS QA Test 853 +# +# parent pointer inject test +# +. ./common/preamble +_begin_fstest auto quick parent + +# get standard environment, filters and checks +. ./common/filter +. ./common/inject +. ./common/parent + +# Modify as appropriate +_supported_fs xfs +_require_scratch +_require_xfs_sysfs debug/larp +_require_xfs_io_error_injection "larp" +_require_xfs_parent +_require_xfs_io_command "parent" + +# real QA test starts here + +# Create a directory tree using a protofile and +# make sure all inodes created have parent pointers + +protofile=$tmp.proto + +cat >$protofile <<EOF +DUMMY1 +0 0 +: root directory +d--777 3 1 +: a directory +testfolder1 d--755 3 1 +$ +: back in the root +testfolder2 d--755 3 1 +: done +$ +EOF + +_scratch_mkfs -f -n parent=1 -p $protofile >>$seqres.full 2>&1 \ + || _fail "mkfs failed" +_check_scratch_fs + +_scratch_mount >>$seqres.full 2>&1 \ + || _fail "mount failed" + +testfolder1="testfolder1" +testfolder2="testfolder2" +file4="file4" +file5="file5" + +echo "" + +# Create files +touch $SCRATCH_MNT/$testfolder1/$file4 +_verify_parent "$testfolder1" "$file4" "$testfolder1/$file4" + +# Inject error +_scratch_inject_error "larp" + +# Move files +mv $SCRATCH_MNT/$testfolder1/$file4 $SCRATCH_MNT/$testfolder2/$file5 2>&1 \ + | _filter_scratch + +# FS should be shut down, touch will fail +touch $SCRATCH_MNT/$testfolder2/$file5 2>&1 | _filter_scratch + +# Remount to replay log +_scratch_remount_dump_log >> $seqres.full + +# FS should be online, touch should succeed +touch $SCRATCH_MNT/$testfolder2/$file5 + +# Check files again +_verify_parent "$testfolder2" "$file5" "$testfolder2/$file5" + +# success, all done +status=0 +exit
tests/xfs/853.out new file +14 -0
@@ -0,0 +1,14 @@ +QA output created by 853 + +*** testfolder1 OK +*** testfolder1/file4 OK +*** testfolder1/file4 OK +*** Verified parent pointer: name:file4, namelen:5 +*** Parent pointer OK for child testfolder1/file4 +mv: cannot stat 'SCRATCH_MNT/testfolder1/file4': Input/output error +touch: cannot touch 'SCRATCH_MNT/testfolder2/file5': Input/output error +*** testfolder2 OK +*** testfolder2/file5 OK +*** testfolder2/file5 OK +*** Verified parent pointer: name:file5, namelen:5 +*** Parent pointer OK for child testfolder2/file5
PATCH 11/14 common/parent: add license and copyright
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Add the necessary licensing and copyright tags to the new file.
--- common/parent | 3 +++ 1 file changed, 3 insertions(+)
common/parent +3 -0
@@ -1,3 +1,6 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022-2023, Oracle and/or its affiliates. All Rights Reserved. # # Parent pointer common functions #
PATCH 12/14 common/parent: don't _fail on missing parent pointer components
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Use echo instead of _fail here so that we run as much of the test as possible. There's no need to stop the test immediately even if the pptr code isn't working.
--- common/parent | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
common/parent +8 -8
@@ -105,21 +105,21 @@ _verify_parent() # Verify parent exists if [ ! -d $SCRATCH_MNT/$parent_path ]; then - _fail "$SCRATCH_MNT/$parent_path not found" + echo "$SCRATCH_MNT/$parent_path not found" else echo "*** $parent_path OK" fi # Verify child exists if [ ! -f $SCRATCH_MNT/$child_path ]; then - _fail "$SCRATCH_MNT/$child_path not found" + echo "$SCRATCH_MNT/$child_path not found" else echo "*** $child_path OK" fi # Verify the parent pointer name exists as a child of the parent if [ ! -f $SCRATCH_MNT/$parent_ppath ]; then - _fail "$SCRATCH_MNT/$parent_ppath not found" + echo "$SCRATCH_MNT/$parent_ppath not found" else echo "*** $parent_ppath OK" fi
@@ -132,7 +132,7 @@ _verify_parent() parents=($($XFS_IO_PROG -x -c \ "parent -f -i $pino -n $parent_pointer_name" $SCRATCH_MNT/$child_path)) if [[ $? != 0 ]]; then - _fail "No parent pointers found for $child_path" + echo "No parent pointers found for $child_path" fi # Parse parent pointer output.
@@ -141,7 +141,7 @@ _verify_parent() # If we didnt find one, bail out if [ $? -ne 0 ]; then - _fail "No parent pointer record found for $parent_path"\ + echo "No parent pointer record found for $parent_path"\ "in $child_path" fi
@@ -150,7 +150,7 @@ _verify_parent() pppino="$(stat -c '%i' $SCRATCH_MNT/$parent_ppath)" if [ $cino -ne $pppino ] then - _fail "Bad parent pointer name value for $child_path."\ + echo "Bad parent pointer name value for $child_path."\ "$SCRATCH_MNT/$parent_ppath belongs to inode $PPPINO,"\ "but should be $cino" fi
@@ -174,7 +174,7 @@ _verify_no_parent() # Verify child exists if [ ! -f $SCRATCH_MNT/$child_path ]; then - _fail "$SCRATCH_MNT/$child_path not found" + echo "$SCRATCH_MNT/$child_path not found" else echo "*** $child_path OK" fi
@@ -195,7 +195,7 @@ _verify_no_parent() return 0 fi - _fail "Parent pointer entry found where none should:"\ + echo "Parent pointer entry found where none should:"\ "inode:$PPINO, gen:$PPGEN," "name:$PPNAME, namelen:$PPNAME_LEN" }
PATCH 13/14 common/parent: check xfs_io parent command paths
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make sure that the paths returned by the xfs_io parent command actually point to the same file.
--- common/parent | 8 ++++++++ 1 file changed, 8 insertions(+)
common/parent +8 -0
@@ -155,6 +155,14 @@ _verify_parent() "but should be $cino" fi + # Make sure path printing works by checking that the paths returned + # all point to the same inode. + local tgt="$SCRATCH_MNT/$child_path" + $XFS_IO_PROG -x -c 'parent -p' "$tgt" | while read pptr_path; do + test "$tgt" -ef "$pptr_path" || \ + echo "$tgt parent pointer $pptr_path should be the same file" + done + echo "*** Verified parent pointer:"\ "name:$PPNAME, namelen:$PPNAME_LEN" echo "*** Parent pointer OK for child $child_path"
PATCH 14/14 xfs/851: test xfs_io parent -p too
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Test the -p argument to the xfs_io parent command too.
--- tests/xfs/851 | 15 +++++++++++++++ tests/xfs/851.out | 10 ++++++++++ 2 files changed, 25 insertions(+)
tests/xfs/851 +15 -0
@@ -12,6 +12,7 @@ _begin_fstest auto quick parent # get standard environment, filters and checks . ./common/parent +. ./common/filter # Modify as appropriate _supported_fs xfs
@@ -96,6 +97,20 @@ ino="$(stat -c '%i' $SCRATCH_MNT/$testfolder2/$file3)" mv -f $SCRATCH_MNT/$testfolder2/$file3 $SCRATCH_MNT/$testfolder1/$file2 _verify_parent "$testfolder1" "$file2" "$testfolder1/$file2" +# Make sure that parent -p filtering works +mkdir -p $SCRATCH_MNT/dira/ $SCRATCH_MNT/dirb/ +dira_inum=$(stat -c '%i' $SCRATCH_MNT/dira) +dirb_inum=$(stat -c '%i' $SCRATCH_MNT/dirb) +touch $SCRATCH_MNT/gorn +ln $SCRATCH_MNT/gorn $SCRATCH_MNT/dira/file1 +ln $SCRATCH_MNT/gorn $SCRATCH_MNT/dirb/file1 +echo look for both +$XFS_IO_PROG -c 'parent -p' $SCRATCH_MNT/gorn | _filter_scratch +echo look for dira +$XFS_IO_PROG -c 'parent -p -n dira' -c "parent -p -i $dira_inum" $SCRATCH_MNT/gorn | _filter_scratch +echo look for dirb +$XFS_IO_PROG -c 'parent -p -n dirb' -c "parent -p -i $dirb_inum" $SCRATCH_MNT/gorn | _filter_scratch + # success, all done status=0 exit
tests/xfs/851.out +10 -0
@@ -57,3 +57,13 @@ QA output created by 851 *** testfolder1/file2 OK *** Verified parent pointer: name:file2, namelen:5 *** Parent pointer OK for child testfolder1/file2 +look for both +SCRATCH_MNT/gorn +SCRATCH_MNT/dira/file1 +SCRATCH_MNT/dirb/file1 +look for dira +SCRATCH_MNT/dira/file1 +SCRATCH_MNT/dira/file1 +look for dirb +SCRATCH_MNT/dirb/file1 +SCRATCH_MNT/dirb/file1
cover xfs: bug fixes for parent pointers
In reply to Darrick J. WongHi all,
This series contains the accumulated bug fixes from Darrick to make fstests pass and online repair work.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-bugfixes
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-bugfixes
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-bugfixes
--- tests/xfs/122.out | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
PATCH 1/1 xfs/122: fix parent pointer ioctl structure sizes
From: Darrick J. Wong <[email protected]>
We've renamed (and resized) the structures for the GETPARENTS ioctl, so adjust the golden output here to reflect that.
--- tests/xfs/122.out | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tests/xfs/122.out +2 -2
@@ -100,6 +100,8 @@ sizeof(struct xfs_fsop_ag_resblks) = 64 sizeof(struct xfs_fsop_geom) = 256 sizeof(struct xfs_fsop_geom_v1) = 112 sizeof(struct xfs_fsop_geom_v4) = 112 +sizeof(struct xfs_getparents) = 96 +sizeof(struct xfs_getparents_rec) = 24 sizeof(struct xfs_icreate_log) = 28 sizeof(struct xfs_inode_log_format) = 56 sizeof(struct xfs_inode_log_format_32) = 52
@@ -110,9 +112,7 @@ sizeof(struct xfs_log_dinode) = 176 sizeof(struct xfs_log_legacy_timestamp) = 8 sizeof(struct xfs_map_extent) = 32 sizeof(struct xfs_parent_name_rec) = 16 -sizeof(struct xfs_parent_ptr) = 280 sizeof(struct xfs_phys_extent) = 16 -sizeof(struct xfs_pptr_info) = 104 sizeof(struct xfs_refcount_key) = 4 sizeof(struct xfs_refcount_rec) = 12 sizeof(struct xfs_rmap_key) = 20
cover fstests: encode parent pointer name in xattr key
In reply to Darrick J. WongHi all,
As I've mentioned in past comments on the parent pointers patchset, the proposed ondisk parent pointer format presents a major difficulty for online directory repair. This difficulty derives from encoding the directory offset of the dirent that the parent pointer is mirroring. Recall that parent pointers are stored in extended attributes:
(parent_ino, parent_gen, diroffset) -> (dirent_name)
If the directory is rebuilt, the offsets of the new directory entries must match the diroffset encoded in the parent pointer, or the filesystem becomes inconsistent. There are a few ways to solve this problem.
One approach would be to augment the directory addname function to take a diroffset and try to create the new entry at that offset. This will not work if the original directory became corrupt and the parent pointers were written out with impossible diroffsets (e.g. overlapping). Requiring matching diroffsets also prevents reorganization and compaction of directories.
This could be remedied by recording the parent pointer diroffset updates necessary to retain consistency, and using the logged parent pointer replace function to rewrite parent pointers as necessary. This is a poor choice from a performance perspective because the logged xattr updates must be committed in the same transaction that commits the new directory structure. If there are a large number of diroffset updates, then the directory commit could take an even longer time.
Worse yet, if the logged xattr updates fill up the transaction, repair will have no choice but to roll to a fresh transaction to continue logging. This breaks repair's policy that repairs should commit atomically. It may break the filesystem as well, since all files involved are pinned until the delayed pptr xattr processing completes. This is a completely bad engineering choice.
Note that the diroffset information is not used anywhere in the directory lookup code. Observe that the only information that we require for a parent pointer is the inverse of an pre-ftype dirent, since this is all we need to reconstruct a directory entry:
(parent_ino, dirent_name) -> NULL
The xattr code supports xattrs with zero-length values, surprisingly. The parent_gen field makes it easy to export parent handle information, so it can be retained:
(parent_ino, parent_gen, dirent_name) -> NULL
Moving the ondisk format to this format is very advantageous for repair code. Unfortunately, there is one hitch: xattr names cannot exceed 255 bytes due to ondisk format limitations. We don't want to constrain the length of dirent names, so instead we could use collision resistant hashes to handle dirents with very long names:
(parent_ino, parent_gen, sha512(dirent_name)) -> (dirent_name)
The first two patches implement this schema. However, this encoding is not maximally efficient, since many directory names are shorter than the length of a sha512 hash. The last three patches in the series bifurcate the parent pointer ondisk format depending on context:
For dirent names shorter than 243 bytes:
(parent_ino, parent_gen, dirent_name) -> NULL
For dirent names longer than 243 bytes:
(parent_ino, parent_gen, dirent_name[0:178],
sha512(child_gen, dirent_name)) -> (dirent_name[179:255])The child file's generation number is mixed into the sha512 computation to make it a little more difficult for unprivileged userspace to attempt collisions.
A messier solution to this problem would be to extend the xattr ondisk format to allow parent pointers to have xattr names up to 267 bytes. This would likely involve redefining the ondisk namelen field to omit the size of the parent ino/gen information and might be madness.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-name-in-attr-key
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-name-in-attr-key
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key
--- tests/xfs/021.out.parent | 22 ++++++++++------------ tests/xfs/122.out | 2 +- 2 files changed, 11 insertions(+), 13 deletions(-)
PATCH 1/1 xfs/{021,122}: adjust parent pointer encoding format
From: Darrick J. Wong <[email protected]>
Adjust these tests to reflect the new parent pointer ondisk format.
--- tests/xfs/021.out.parent | 22 ++++++++++------------ tests/xfs/122.out | 2 +- 2 files changed, 11 insertions(+), 13 deletions(-)
tests/xfs/021.out.parent +10 -12
@@ -19,12 +19,11 @@ size of attr value = 65536 *** unmount FS *** dump attributes (1) -a.sfattr.hdr.totsize = 53 +a.sfattr.hdr.totsize = 49 a.sfattr.hdr.count = 3 -a.sfattr.list[0].namelen = 16 -a.sfattr.list[0].valuelen = 10 +a.sfattr.list[0].namelen = 22 +a.sfattr.list[0].valuelen = 0 a.sfattr.list[0].root = 0 -a.sfattr.list[0].value = "testfile.1" a.sfattr.list[1].namelen = 2 a.sfattr.list[1].valuelen = 3 a.sfattr.list[1].root = 0
@@ -40,7 +39,7 @@ hdr.info.forw = 0 hdr.info.back = 0 hdr.info.magic = 0xfbee hdr.count = 4 -hdr.usedbytes = 84 +hdr.usedbytes = 80 hdr.firstused = FIRSTUSED hdr.holes = 0 hdr.freemap[0-2] = [base,size] [FREEMAP..]
@@ -53,12 +52,11 @@ nvlist[1].valueblk = 0x1 nvlist[1].valuelen = 65535 nvlist[1].namelen = 2 nvlist[1].name = "a3" -nvlist[2].valuelen = 10 -nvlist[2].namelen = 16 -nvlist[2].value = "testfile.2" -nvlist[3].valuelen = 8 -nvlist[3].namelen = 7 -nvlist[3].name = "a2-----" -nvlist[3].value = "value_2\d" +nvlist[2].valuelen = 8 +nvlist[2].namelen = 7 +nvlist[2].name = "a2-----" +nvlist[2].value = "value_2\d" +nvlist[3].valuelen = 0 +nvlist[3].namelen = 22 *** done *** unmount
tests/xfs/122.out +1 -1
@@ -111,7 +111,7 @@ sizeof(struct xfs_legacy_timestamp) = 8 sizeof(struct xfs_log_dinode) = 176 sizeof(struct xfs_log_legacy_timestamp) = 8 sizeof(struct xfs_map_extent) = 32 -sizeof(struct xfs_parent_name_rec) = 16 +sizeof(struct xfs_parent_name_rec) = 12 sizeof(struct xfs_phys_extent) = 16 sizeof(struct xfs_refcount_key) = 4 sizeof(struct xfs_refcount_rec) = 12
Quoted 70 lines · Darrick J. Wong
Hi everyone,
This deluge contains all of the additions to the parent pointers patchset that I've been working since last month's deluge. The kernel
and xfsprogs patchsets are based on Allison's v10 tag from last week; the fstests patches are merely a part of my development tree. To recap
Allison's cover letter:"The goal of this patch set is to add a parent pointer attribute to each
inode. The attribute name containing the parent inode, generation, and
directory offset, while the attribute value contains the file name. This feature will enable future optimizations for online scrub, shrink,
nfs handles, verity, or any other feature that could make use of quickly
deriving an inodes path from the mount point."v10r1d2 rebases everything against 6.3-rc2. I still want to remove the
diroffset from the ondisk parent pointer, but for v10 I've replaced the
sha512 hashing code with modifications to the xattr code to support lookups based on name *and* value. With that working, we can encode parent pointers like this:(parent_ino,
parent_gen, name[])xattr lookups still work correctly, and repair doesn't have to deal with
keeping the diroffsets in sync if the directory gets rebuilt. With this
change applied, I'm ready to weave my new changes into Allison's v10 and
call parent pointers done. :)The online directory and parent pointer code are exactly the same as the
v9r2d1 release, so I'm eliding that and everything that was in Allison's
recent v10 patchset. IOWs, this deluge includes only the bug fixes I've
made to parent pointers, the updates I've made to the ondisk format, and
the necessary changes to fstests to get everything to pass.If you want to pull the whole thing, use these links:
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSrGWwDFp$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSmjOh6X7$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSlulOhuJ$Allison: Could you please resync libxfs in the following patches under
https://urldefense.com/v3/__https://github.com/allisonhenderson/xfsprogs/commits/xfsprogs_new_pptrs_v10__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSqiXa3xN$
please?xfsprogs: add parent pointer support to attribute code
xfsprogs: extend transaction reservations for parent attributes
xfsprogs: parent pointer attribute creation
xfsprogs: remove parent pointers in unlink
xfsprogs: Add parent pointers to rename
xfsprogs: move/add parent pointer validators toxfs_parent
Quoted 5 lines
There are discrepancies between the two, which makes
./tools/libxfs-diff
unhappy. Or, if you want me to merge my ondisk format changes into my
branches, I'll put out v11 with everything taken care of.
Sure, will resync, as I recall some of them had to deviate a little bit because the corresponding code appears in different places, or needed special handling.
Originally my intent was just to get the kernel side of things settled and landed first, and then grind through the other spaces since user space is mostly a port. I was trying to avoid sending out giant deluges since people seemed to get hung up enough in just kernel space reviews.
Thanks for all the help tho.
Allison
Quoted 2 lines
--D
Quoted 71 lines · Allison Henderson
Quoted 70 lines · Darrick J. Wong
Hi everyone,
This deluge contains all of the additions to the parent pointers patchset that I've been working since last month's deluge. The kernel
and xfsprogs patchsets are based on Allison's v10 tag from last week; the fstests patches are merely a part of my development tree. To recap
Allison's cover letter:"The goal of this patch set is to add a parent pointer attribute to each
inode. The attribute name containing the parent inode, generation, and
directory offset, while the attribute value contains the file name. This feature will enable future optimizations for online scrub, shrink,
nfs handles, verity, or any other feature that could make use of quickly
deriving an inodes path from the mount point."v10r1d2 rebases everything against 6.3-rc2. I still want to remove the
diroffset from the ondisk parent pointer, but for v10 I've replaced the
sha512 hashing code with modifications to the xattr code to support lookups based on name *and* value. With that working, we can encode parent pointers like this:(parent_ino,
parent_gen, name[])xattr lookups still work correctly, and repair doesn't have to deal with
keeping the diroffsets in sync if the directory gets rebuilt. With this
change applied, I'm ready to weave my new changes into Allison's v10 and
call parent pointers done. :)The online directory and parent pointer code are exactly the same as the
v9r2d1 release, so I'm eliding that and everything that was in Allison's
recent v10 patchset. IOWs, this deluge includes only the bug fixes I've
made to parent pointers, the updates I've made to the ondisk format, and
the necessary changes to fstests to get everything to pass.If you want to pull the whole thing, use these links:
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSrGWwDFp$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSmjOh6X7$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSlulOhuJ$Allison: Could you please resync libxfs in the following patches under
https://urldefense.com/v3/__https://github.com/allisonhenderson/xfsprogs/commits/xfsprogs_new_pptrs_v10__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSqiXa3xN$
please?xfsprogs: add parent pointer support to attribute code
xfsprogs: extend transaction reservations for parent attributes
xfsprogs: parent pointer attribute creation
xfsprogs: remove parent pointers in unlink
xfsprogs: Add parent pointers to rename
xfsprogs: move/add parent pointer validators toxfs_parent
Quoted 8 lines
Quoted 5 lines
There are discrepancies between the two, which makes
./tools/libxfs-diff
unhappy. Or, if you want me to merge my ondisk format changes into my
branches, I'll put out v11 with everything taken care of.Sure, will resync, as I recall some of them had to deviate a little bit because the corresponding code appears in different places, or needed special handling.
Ok, thank you. It's easier to develop xfsprogs code when libxfs can be kept in sync easily.
Quoted 5 lines
Originally my intent was just to get the kernel side of things settled and landed first, and then grind through the other spaces since user space is mostly a port. I was trying to avoid sending out giant deluges since people seemed to get hung up enough in just kernel space reviews.
Yeah, 'tis true that sending xfsprogs libxfs patches prematurely is just noise on the list. :/
That said, the closer a patchset gets to final review, the tidier the xfsprogs part ought to be. I've added support in repair and the debugger, which means this is /very/ close to its final review.
--D
Quoted 7 lines
Thanks for all the help tho.
Allison
Quoted 2 lines
--D
Quoted 99 lines · Darrick J. Wong
Quoted 92 lines · Allison Henderson
Quoted 84 lines · Darrick J. Wong
Hi everyone,
This deluge contains all of the additions to the parent pointers patchset that I've been working since last month's deluge. The kernel
and xfsprogs patchsets are based on Allison's v10 tag from last week;
the fstests patches are merely a part of my development tree. To recap
Allison's cover letter:"The goal of this patch set is to add a parent pointer attribute to
each
inode. The attribute name containing the parent inode, generation,
and
directory offset, while the attribute value contains the file name.
This feature will enable future optimizations for online scrub, shrink,
nfs handles, verity, or any other feature that could make use of quickly
deriving an inodes path from the mount point."v10r1d2 rebases everything against 6.3-rc2. I still want to remove
the
diroffset from the ondisk parent pointer, but for v10 I've replaced
the
sha512 hashing code with modifications to the xattr code to support
lookups based on name *and* value. With that working, we can encode
parent pointers like this:(parent_ino,
parent_gen, name[])xattr lookups still work correctly, and repair doesn't have to deal
with
keeping the diroffsets in sync if the directory gets rebuilt. With
this
change applied, I'm ready to weave my new changes into Allison's v10
and
call parent pointers done. :)The online directory and parent pointer code are exactly the same as
the
v9r2d1 release, so I'm eliding that and everything that was in Allison's
recent v10 patchset. IOWs, this deluge includes only the bug fixes
I've
made to parent pointers, the updates I've made to the ondisk format,
and
the necessary changes to fstests to get everything to pass.If you want to pull the whole thing, use these links:
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSrGWwDFp$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSmjOh6X7$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSlulOhuJ$Allison: Could you please resync libxfs in the following patches under
https://urldefense.com/v3/__https://github.com/allisonhenderson/xfsprogs/commits/xfsprogs_new_pptrs_v10__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSqiXa3xN$
please?xfsprogs: add parent pointer support to attribute code
xfsprogs: extend transaction reservations for parent attributes
xfsprogs: parent pointer attribute creation
xfsprogs: remove parent pointers in unlink
xfsprogs: Add parent pointers to rename
xfsprogs: move/add parent pointer validators toxfs_parentQuoted 7 lines
There are discrepancies between the two, which makes
./tools/libxfs-
diff
unhappy. Or, if you want me to merge my ondisk format changes into
my
branches, I'll put out v11 with everything taken care of.Quoted 5 lines
Sure, will resync, as I recall some of them had to deviate a little bit
because the corresponding code appears in different places, or needed
special handling.
Quoted 3 lines
Ok, thank you. It's easier to develop xfsprogs code when libxfs can be
kept in sync easily.
Here is a resync, some of them still needed hand porting, but it looks like it sync'd up a lot of white space so let me know if this clears up things for you
https://github.com/allisonhenderson/xfsprogs/tree/xfsprogs_new_pptrs_v10r1
Allison
Quoted 27 lines
Quoted 9 lines
Originally my intent was just to get the kernel side of things settled
and landed first, and then grind through the other spaces since user
space is mostly a port. I was trying to avoid sending out giant deluges since people seemed to get hung up enough in just kernel space
reviews.Yeah, 'tis true that sending xfsprogs libxfs patches prematurely is just
noise on the list. :/That said, the closer a patchset gets to final review, the tidier the xfsprogs part ought to be. I've added support in repair and the debugger, which means this is /very/ close to its final review.
--D
Quoted 7 lines
Thanks for all the help tho.
Allison
Quoted 2 lines
--D
Quoted 100 lines · Allison Henderson
Quoted 99 lines · Darrick J. Wong
Quoted 97 lines · Allison Henderson
Quoted 84 lines · Darrick J. Wong
Hi everyone,
This deluge contains all of the additions to the parent pointers patchset that I've been working since last month's deluge. The kernel
and xfsprogs patchsets are based on Allison's v10 tag from last week;
the fstests patches are merely a part of my development tree. To recap
Allison's cover letter:"The goal of this patch set is to add a parent pointer attribute to
each
inode. The attribute name containing the parent inode, generation,
and
directory offset, while the attribute value contains the file name.
This feature will enable future optimizations for online scrub, shrink,
nfs handles, verity, or any other feature that could make use of quickly
deriving an inodes path from the mount point."v10r1d2 rebases everything against 6.3-rc2. I still want to remove
the
diroffset from the ondisk parent pointer, but for v10 I've replaced
the
sha512 hashing code with modifications to the xattr code to support
lookups based on name *and* value. With that working, we can encode
parent pointers like this:(parent_ino,
parent_gen, name[])xattr lookups still work correctly, and repair doesn't have to deal
with
keeping the diroffsets in sync if the directory gets rebuilt. With
this
change applied, I'm ready to weave my new changes into Allison's v10
and
call parent pointers done. :)The online directory and parent pointer code are exactly the same as
the
v9r2d1 release, so I'm eliding that and everything that was in Allison's
recent v10 patchset. IOWs, this deluge includes only the bug fixes
I've
made to parent pointers, the updates I've made to the ondisk format,
and
the necessary changes to fstests to get everything to pass.If you want to pull the whole thing, use these links:
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSrGWwDFp$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=pptrs-drop-unnecessary__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSmjOh6X7$
https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=pptrs-name-in-attr-key__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSlulOhuJ$Allison: Could you please resync libxfs in the following patches under
https://urldefense.com/v3/__https://github.com/allisonhenderson/xfsprogs/commits/xfsprogs_new_pptrs_v10__;!!ACWV5N9M2RV99hQ!MnkBbyDKEdgQiXLfmXZ87uT_j_TAtQHHA1UraPf01op6wNpRZkk2tg5CXru4eL6-pzJyUl-uJAZlSqiXa3xN$
please?xfsprogs: add parent pointer support to attribute code
xfsprogs: extend transaction reservations for parent attributes
xfsprogs: parent pointer attribute creation
xfsprogs: remove parent pointers in unlink
xfsprogs: Add parent pointers to rename
xfsprogs: move/add parent pointer validators toxfs_parentQuoted 7 lines
There are discrepancies between the two, which makes
./tools/libxfs-
diff
unhappy. Or, if you want me to merge my ondisk format changes into
my
branches, I'll put out v11 with everything taken care of.Sure, will resync, as I recall some of them had to deviate a little bit
because the corresponding code appears in different places, or needed
special handling.
Quoted 8 lines
Quoted 3 lines
Ok, thank you. It's easier to develop xfsprogs code when libxfs can be
kept in sync easily.Here is a resync, some of them still needed hand porting, but it looks like it sync'd up a lot of white space so let me know if this clears up things for you
https://github.com/allisonhenderson/xfsprogs/tree/xfsprogs_new_pptrs_v10r1
Yep, thanks. I'll incorporate that into v11. :)
--D
Quoted 30 lines
Allison
Quoted 27 lines
Quoted 9 lines
Originally my intent was just to get the kernel side of things settled
and landed first, and then grind through the other spaces since user
space is mostly a port. I was trying to avoid sending out giant deluges since people seemed to get hung up enough in just kernel space
reviews.Yeah, 'tis true that sending xfsprogs libxfs patches prematurely is just
noise on the list. :/That said, the closer a patchset gets to final review, the tidier the xfsprogs part ought to be. I've added support in repair and the debugger, which means this is /very/ close to its final review.
--D
Quoted 7 lines
Thanks for all the help tho.
Allison
Quoted 2 lines
--D