xfsprogs: v3.2.0 released!

Hello!

It is my pleasure to announce the release of v3.2.0 of the xfsprogs package. This release has been a long time in the making with a lot of effort from many people:

- just over a year since v3.1.11 was released
- 323 commits
- 17 contributors
- 10 code reviewers
- 232 files changed
- +29549/-24432 lines of code

The major feature that this release brings is support for the version 5 on-disk format. This new format provides significant reliability enhancements such as metadata CRCs, object back owner references and improved crash recovery reliability.

The next kernel to be released (v3.15) will also be the first kernel to support this new on-disk format for production workloads: the kernel no longer considers this new format experimental as of v3.15-rc5.

There are many other enhancements:

- xfs_repair has significantly improved multithreading scalability
- xfs_db now uses the common libxfs IO engine
- xfs_db validates the metadata it reads from disk
- xfs_io has support for a wide range of new kernel features
- xfs_metadump has much improved obfuscation support
- libxfs has significanty fewer differences to the kernel code it is derived from

The source code can be accessed via git using this URL:

git://oss.sgi.com/xfs/cmds/xfsprogs.git

A signed gzipped-tar archive of the source code is available here:

	ftp://oss.sgi.com/projects/xfs/cmd_tars/xfsprogs-3.2.0.tar.gz
	ftp://oss.sgi.com/projects/xfs/cmd_tars/xfsprogs-3.2.0.tar.gz.sign

Problems, issues, questions and general discussion about the release should be directed to the XFS mailing list ([email protected]).

A summary of the changes during development of this release (taken from doc/CHANGES) is as follows:

xfsprogs-3.2.0 (16 May 2014)
- First release with full support of CRC enabled filesystems
- No code changes from 3.2.0-rc3

xfsprogs-3.2.0-rc3 (9 May 2014)
- Third release candidate for full support of CRC enabled filesystems
- Updated Debian change logs in preparation for release (Nathan Scott)
- Build warning fixes (Nathan Scott)
- xfs_repair prefetch fix (Eric Sandeen)
- xfs_repair block tracking scalability fix

xfsprogs-3.2.0-rc2 (2 May 2014)
- Second release candidate for full support of CRC enabled filesystems
- xfs_repair has full CRC validation and repair
- Coverity related cleanups and fixes

xfsprogs-3.2.0-rc1 (14 April 2014)
- First release candidate for full support of CRC enabled filesystems
- Large number of Coverity related fixes and cleanups
- disambiguous of CRC validation errors from IO errors.
- Improved dangerous mode handling in repair
- repair handles garbage in zeroed areas of superblocks better
- repair validates dirent ftype field fully
- metadump fully supports discontiguous directory blocks
- metadump only recalculates CRCs on metadata it obfuscates so as to preserve errors in the metadata where possible.
- default log size that mkfs creates is now reverted to the same size as 3.1.x releases create.
- mkfs sets the ftype on directory entries correctly during protofile population
- xfs_io support O_TMPFILE, flink, FALLOC_FL_ZERO_RANGE and FALLOC_FL_COLLAPSE_RANGE,
- logprint handles split entries better

xfsprogs-3.2.0-alpha2 (25 November 2013)
- Alpha release for the purpose of testing the CRC feature in kernels 3.10 and newer.
- Enable xfs_db write support and xfs_metadump support for CRC enabled filesystems.
- Add directory entry filetype support for non-CRC filesystems.
- Remove experimental warnings for CRC filesystems.
- Ensure all inodes created by xfs_repair have a proper d_type set.
- Fix build on big endian machines.
- Properly handle symlinks to devices on various tool commandlines.
- Fix xfs_repair's dirty log detection for 4k sector logs, broken in Alpha1.
- Fix a potential segfault in xfs_repair when issuing progress reports.
- Fix potential xfs_fsr failures when running w/ selinux.
- Update config.guess/config.sub for arm64, thanks to Colin Watson.
- Stop wasting memory by caching inode structures in xfs_repair - they are never re-used. Thanks to Christoph Hellwig.
- Fix several Coverity-found defects, thanks to Li Zhong.
- Fix platform_test_xfs_fd to return false on special files which cannot take an xfs ioctl.
- Sync up libxfs with kernel code.
- Improved xfs_repair performance on large filesystems (always use prefetch and strided AG scanning functionality)

xfsprogs-3.2.0-alpha1 (26 September 2013)
- Alpha release for the purpose of testing the CRC feature in kernels 3.10 and newer.
- Remove all vestiges of old, unsupported version 1 directory code.
- Add a "readdir" command to xfs_io, thanks to Brian Foster.
- Fix potential segfault in xfs_repair when creating lost+found.
- Zero out unused parts of on-disk superblocks during repair, to avoid metadata verifier failures at runtime.
- Add directory entry type support to mkfs.xfs and xfs_db.
- Add the icreate transaction to xfs_logprint, and fix continuation transactions.
- Add the lseek SEEK_DATA/SEEK_HOLE support into xfs_io.
- Print all AGI unlinked buckets in xfs_logprint.
- Fix mkfs.xfs ENOSPC with protofile which creates a very large directory.
- Fix several Coverity-found defects, thanks to Li Zhong.
- Do all file reads in xfs_fsr using O_DIRECT.
- Sync up libxfs with kernel code.
- Add support for concurrent group and project quota usage on CRC enabled filesystems.
- Ensure mkfs creates log sizes that are always large enough for the configured fileystem geometry.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

If this is built with DEBUG= (i.e. not defaulting to DEBUG=-NDEBUG), several source files fail to compile -- it looks like there are a number of assertions that haven't been updated for changes in the code:

logitem.c:53:24: error: 'struct xfs_buf' has no member named 'b_map_count'

    ASSERT(blip->bli_buf->b_map_count == nmaps);
                        ^
trans.c:52:12: error: 'struct xfs_log_item' has no member named 'li_ailp'
  ASSERT(lip->li_ailp == tp->t_mountp->m_ail);
            ^
trans.c:52:37: error: 'xfs_mount_t' has no member named 'm_ail'
  ASSERT(lip->li_ailp == tp->t_mountp->m_ail);
                                     ^
util.c:253:40: error: 'mp' undeclared (first use in this function)
   ASSERT(uuid_equal(&ip->i_d.di_uuid, &mp->m_sb.sb_uuid));
                                        ^
util.c:401:33: error: 'xfs_ifork_t' has no member named 'if_ext_max'
   ip->i_d.di_nextents > ip->i_df.if_ext_max);
                                 ^

xfs_btree.c:545:2: warning: implicit declaration of function 'xfs_buf_geterror' [-Wimplicit-function-declaration]
ASSERT(!xfs_buf_geterror(bp));
^
xfs_btree.c:2052:21: error: 'const struct xfs_btree_ops' has no member named 'keys_inorder'
ASSERT(cur->bc_ops->keys_inorder(cur,

                     ^
xfs_btree.c:2064:21: error: 'const struct xfs_btree_ops' has no member named 'recs_inorder'
   ASSERT(cur->bc_ops->recs_inorder(cur,
                     ^
xfs_btree.c:2240:21: error: 'const struct xfs_btree_ops' has no member named 'keys_inorder'
   ASSERT(cur->bc_ops->keys_inorder(cur, rkp,
                     ^
xfs_btree.c:2259:21: error: 'const struct xfs_btree_ops' has no member named 'recs_inorder'
   ASSERT(cur->bc_ops->recs_inorder(cur, rrp,
                     ^
xfs_dir2_node.c:1053:36: error: 'oldstale' undeclared (first use in this function)
  ASSERT(hdr1.stale + hdr2.stale == oldstale);
                                    ^

xfs_inode_fork.c:728:2: warning: implicit declaration of function 'xfs_isilocked' [-Wimplicit-function-declaration]
ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
^
xfs_inode_fork.c:728:42: error: 'XFS_ILOCK_SHARED' undeclared (first use in this function)

  ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
                                          ^

(Errors from current Git, built with GCC 4.9. You also get a good crop of warnings if you build it with clang 3.4.1's scan-build, which'd be worth checking out in case there's anything serious there.)

It's signed with a different key from the previous release -- it'd be useful to mention the key ID in the announcement so that it can be verified.

Cheers,

Adam Sampson <[email protected]>                         <http://offog.org/>

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Can't say I've ever built xfsprogs with "DEBUG=". I'm not sure there's really any benefit in doing so - it's preferable to have things like xfs_repair abort when it comes across an inconsistency it can't handle than to continue blindly along and making a bigger mess of the filesystem it's supposed to be fixing...

Anyway, we'll look to fix it for 3.2.1.

ISTR that was done recently by Eric, and I've run clang recently, too.

Should be obvious, yes? Especially with the release announcement being signed, too. As it is, release tarballs have always been signed with the current maintainer's key, so when the maintainer changes so does the signing key....

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

-NDEBUG is only default for libxfs/ AFAIK:

# grep -B1 NDEBUG libxfs/Makefile 
# don't try linking xfs_repair with a debug libxfs.
DEBUG = -DNDEBUG

For everything else, default should be -DDEBUG:

configure: DEBUG=${DEBUG:-'-DDEBUG'} debug_build="$DEBUG"

so default is -DDEBUG for everything except libxfs, with -NDEBUG; I'm not sure what '' does - break, apparently. Seems like maybe just a bit of a makefile mess and maybe old crufty code. Patches accepted and all that. :)

But maybe it's time to remove it as an option, and just DTRT everywhere.

Some, yeah. I don't claim to have gotten it all cleaned up yet, though. We also keep track of what Coverity finds, chipping away at a backlog of potential defects...

Thanks for the report, though, there's always more to do...

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Yes -- that's why I was building with DEBUG= on previous releases (i.e. I want assertions enabled). doc/INSTALL says that DEBUG=-DNDEBUG disables assertions, so packagers are quite likely to have DEBUG= in their build process.

Cheers. :)

Adam Sampson <[email protected]>                         <http://offog.org/>

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hmmm - so, not being an everyday userspace programmer, it didn't even occur to me that "-DNDEBUG" actually changes libc header behaviour, not anything to do with the XFS code.

$ man assert
.....
BUGS
       assert()  is  implemented as a macro; if the expression
       tested has side-effects, program behavior will be different
       depending on whether NDEBUG is defined.  This may create
       Heisenbugs which go away when debugging is turned on.

Yup, it's oh so obvious now that "NDEBUG" is something owned by system library code, not the xfsprogs package...

<sigh>

Or maybe not. The intent of always turning off the asserts is that code like xfs_repair shouldn't assert fail when stuff it detected as out of bounds in a library function. IOWs, you're quite likely to unintentionally break repair by removing the NDEBUG define to re-instate the library asserts...

The control of the assert statements in the xfs_repair code itself is handled by the -DDEBUG macro, which is configurable. i.e. you can chose whether or not to have asserts in the repair code itself for it to fail when an inconsistency it can't handle is detected, but repair defines "inconsistent and cannot continue" very differently to the libxfs library code....

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

And it's not straight forward, either, because some of the ASSERT()s have code in them that is only present when the libxfs code is built with -DDEBUG, so compiling in the asserts without enabling -DDEBUG is not going to work. And, well, compiling with -DDEBUG itself deosn't work nor with -DNDEBUG -DDEBUG because -DDEBUG enables code that has unhandled kernelisms in it.

So, right now, don't remove -NDEBUG from the libxfs/Makefile. It's there for a good reason right now. i.e. removing it requires some non-trivial work, and even if we do make it compile the resultant code will probably not work (i.e. assert fail unnecessarily) given the requirements that userspace has for tolerance of inconsistent on-disk states.

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs