Skip to contentSkip to conversation

libxlog: fix log buffer alignment

PATCH Dave Chinner 2 participants 7 messages May 1, 2013 – May 9, 2013 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20130430121300.GB10481@dastard

#1 lore raw

From: Dave Chinner <[email protected]>

The libxlog sync changed the way log buffers used in recovery are aligned to sector sizes. The old code used to check for a zero log sector size and not do any alignment - this check was removed from the kernel code because there is always log sector size configured. It turns out that userspace *never* sets the log sector size, so userspace has always simply ignored the alignment in log replay (oops!).

This problem manifested itself as detecting a freshly mkfs'd filesystem as having a totally zeroed log and hence xfs_check was emitting output and hence xfstests was refusing to run.

This patch correctly configures the callers that use the log recovery code so that the log sector size is set appropriately and ensures that log recovery operates as expected.

---
 db/sb.c                   |    5 +++++
 libxlog/xfs_log_recover.c |    5 ++++-
 logprint/logprint.c       |    6 ++++++
 repair/phase2.c           |    4 ++++
 4 files changed, 19 insertions(+), 1 deletion(-)
db/sb.c +5 -0
index d83db9c..4da1f6a 100644
@@ -235,10 +235,15 @@ sb_logcheck(void)
 		x.logdev = x.ddev;
 	x.logBBsize = XFS_FSB_TO_BB(mp, mp->m_sb.sb_logblocks);
 	x.logBBstart = XFS_FSB_TO_DADDR(mp, mp->m_sb.sb_logstart);
+	x.lbsize = BBSIZE;
+	if (xfs_sb_version_hassector(&mp->m_sb))
+		x.lbsize <<= (mp->m_sb.sb_logsectlog - BBSHIFT);
+
 	log.l_dev = (mp->m_sb.sb_logstart == 0) ? x.logdev : x.ddev;
 	log.l_logsize = BBTOB(log.l_logBBsize);
 	log.l_logBBsize = x.logBBsize;
 	log.l_logBBstart = x.logBBstart;
+	log.l_sectBBsize  = BTOBB(x.lbsize);
 	log.l_mp = mp;
 
 	if (xlog_find_tail(&log, &head_blk, &tail_blk)) {
libxlog/xfs_log_recover.c +4 -1
index ad53e86..028d1e3 100644
@@ -21,7 +21,10 @@
 #define xfs_readonly_buftarg(buftarg)			(0)
 
 /* avoid set-but-unused var warning. gcc is not very bright. */
-#define xlog_clear_stale_blocks(log, taillsn)		((taillsn) = (taillsn))
+#define xlog_clear_stale_blocks(log, taillsn)		({ \
+	(taillsn) = (taillsn); \
+	(0); \
+})
 
 
 /*
logprint/logprint.c +6 -0
index 813424a..3fbcdba 100644
@@ -94,6 +94,10 @@ logstat(xfs_mount_t *mp)
 
 		x.logBBsize = XFS_FSB_TO_BB(mp, sb->sb_logblocks);
 		x.logBBstart = XFS_FSB_TO_DADDR(mp, sb->sb_logstart);
+		x.lbsize = BBSIZE;
+		if (xfs_sb_version_hassector(sb))
+			x.lbsize <<= (sb->sb_logsectlog - BBSHIFT);
+
 		if (!x.logname && sb->sb_logstart == 0) {
 			fprintf(stderr, _("    external log device not specified\n\n"));
 			usage();
@@ -105,6 +109,7 @@ logstat(xfs_mount_t *mp)
 		stat(x.dname, &s);
 		x.logBBsize = s.st_size >> 9;
 		x.logBBstart = 0;
+		x.lbsize = BBSIZE;
 	}
 
 
@@ -235,6 +240,7 @@ main(int argc, char **argv)
 	log.l_logsize     = BBTOB(x.logBBsize);
 	log.l_logBBstart  = x.logBBstart;
 	log.l_logBBsize   = x.logBBsize;
+	log.l_sectBBsize  = BTOBB(x.lbsize);
 	log.l_mp          = &mount;
 
 	switch (print_operation) {
repair/phase2.c +4 -0
index 23b457a..382cd7b 100644
@@ -47,11 +47,15 @@ zero_log(xfs_mount_t *mp)
 		x.logdev = x.ddev;
 	x.logBBsize = XFS_FSB_TO_BB(mp, mp->m_sb.sb_logblocks);
 	x.logBBstart = XFS_FSB_TO_DADDR(mp, mp->m_sb.sb_logstart);
+	x.lbsize = BBSIZE;
+	if (xfs_sb_version_hassector(&mp->m_sb))
+		x.lbsize <<= (mp->m_sb.sb_logsectlog - BBSHIFT);
 
 	log.l_dev = logdev;
 	log.l_logsize = BBTOB(x.logBBsize);
 	log.l_logBBsize = x.logBBsize;
 	log.l_logBBstart = x.logBBstart;
+	log.l_sectBBsize  = BTOBB(x.lbsize);
 	log.l_mp = mp;
 	if (xfs_sb_version_hassector(&mp->m_sb)) {
 		log.l_sectbb_log = mp->m_sb.sb_logsectlog - BBSHIFT;

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#2 lore raw
Quoted 14 lines · Dave Chinner

From: Dave Chinner<[email protected]>

The libxlog sync changed the way log buffers used in recovery are aligned to sector sizes. The old code used to check for a zero log sector size and not do any alignment - this check was removed from the kernel code because there is always log sector size configured. It turns out that userspace *never* sets the log sector size, so userspace has always simply ignored the alignment in log replay (oops!).

This problem manifested itself as detecting a freshly mkfs'd filesystem as having a totally zeroed log and hence xfs_check was emitting output and hence xfstests was refusing to run.

Quoted 6 lines

This patch correctly configures the callers that use the log recovery code so that the log sector size is set appropriately and ensures that log recovery operates as expected.

---

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw

[PATCH 10/8] xfsdb: re-instate DA btree node headers

In reply to a message not in the archive

From: Dave Chinner <[email protected]>

When removing the dirv1 code, it wasn't immediately obvious that the dir v2 code used a small chunk of the dirv1 field definitions. i.e. those for the DA btree node headers. Hence bits of xfs_db didn't work as expected, and some tests failed in non-obvious ways. e.g test 073 failed with this additional line of output:

Use of uninitialized value $logstart in numeric gt (>) at /home/dave/src/xfstests-dev/src/fill2fs line 84

which was the result of the command:

xfs_db -r -c sb -c $TEST_DEV

giving an unexpectedly incorrect output.

Re-instate the needed field definitions and rename them from "DIR" to "DA" so it is obvious they are for decoding DA Btree format blocks.

---
 db/dir2.c  |   30 +++++++++++++++++++++++++++---
 db/dir2.h  |    4 ++++
 db/field.c |    7 +++++++
 db/field.h |   15 +++------------
 4 files changed, 41 insertions(+), 15 deletions(-)
db/dir2.c +27 -3
index a13fa25..a539f2d 100644
@@ -84,9 +84,9 @@ const field_t	dir2_flds[] = {
 	  FLD_ARRAY|FLD_COUNT, TYP_NONE },
 	{ "ltail", FLDT_DIR2_LEAF_TAIL, dir2_leaf_tail_offset,
 	  dir2_leaf_tail_count, FLD_OFFSET|FLD_COUNT, TYP_NONE },
-	{ "nhdr", FLDT_DIR_NODE_HDR, OI(NOFF(hdr)), dir2_node_hdr_count,
+	{ "nhdr", FLDT_DA_NODE_HDR, OI(NOFF(hdr)), dir2_node_hdr_count,
 	  FLD_COUNT, TYP_NONE },
-	{ "nbtree", FLDT_DIR_NODE_ENTRY, OI(NOFF(btree)), dir2_node_btree_count,
+	{ "nbtree", FLDT_DA_NODE_ENTRY, OI(NOFF(btree)), dir2_node_btree_count,
 	  FLD_ARRAY|FLD_COUNT, TYP_NONE },
 	{ "fhdr", FLDT_DIR2_FREE_HDR, OI(FOFF(hdr)), dir2_free_hdr_count,
 	  FLD_COUNT, TYP_NONE },
@@ -144,7 +144,7 @@ const field_t	dir2_leaf_entry_flds[] = {
 
 #define	LHOFF(f)	bitize(offsetof(xfs_dir2_leaf_hdr_t, f))
 const field_t	dir2_leaf_hdr_flds[] = {
-	{ "info", FLDT_DIR_BLKINFO, OI(LHOFF(info)), C1, 0, TYP_NONE },
+	{ "info", FLDT_DA_BLKINFO, OI(LHOFF(info)), C1, 0, TYP_NONE },
 	{ "count", FLDT_UINT16D, OI(LHOFF(count)), C1, 0, TYP_NONE },
 	{ "stale", FLDT_UINT16D, OI(LHOFF(stale)), C1, 0, TYP_NONE },
 	{ NULL }
@@ -165,6 +165,30 @@ const field_t	dir2_free_hdr_flds[] = {
 	{ NULL }
 };
 
+#define	DBOFF(f)	bitize(offsetof(xfs_da_blkinfo_t, f))
+const field_t	da_blkinfo_flds[] = {
+	{ "forw", FLDT_DIRBLOCK, OI(DBOFF(forw)), C1, 0, TYP_INODATA },
+	{ "back", FLDT_DIRBLOCK, OI(DBOFF(back)), C1, 0, TYP_INODATA },
+	{ "magic", FLDT_UINT16X, OI(DBOFF(magic)), C1, 0, TYP_NONE },
+	{ "pad", FLDT_UINT16X, OI(DBOFF(pad)), C1, FLD_SKIPALL, TYP_NONE },
+	{ NULL }
+};
+
+#define	EOFF(f)	bitize(offsetof(xfs_da_node_entry_t, f))
+const field_t	da_node_entry_flds[] = {
+	{ "hashval", FLDT_UINT32X, OI(EOFF(hashval)), C1, 0, TYP_NONE },
+	{ "before", FLDT_DIRBLOCK, OI(EOFF(before)), C1, 0, TYP_INODATA },
+	{ NULL }
+};
+
+#define	HOFF(f)	bitize(offsetof(xfs_da_node_hdr_t, f))
+const field_t	da_node_hdr_flds[] = {
+	{ "info", FLDT_DA_BLKINFO, OI(HOFF(info)), C1, 0, TYP_NONE },
+	{ "count", FLDT_UINT16D, OI(HOFF(count)), C1, 0, TYP_NONE },
+	{ "level", FLDT_UINT16D, OI(HOFF(level)), C1, 0, TYP_NONE },
+	{ NULL }
+};
+
 /*ARGSUSED*/
 static int
 dir2_block_hdr_count(
db/dir2.h +4 -0
index 5e117ac..a5f0bec 100644
@@ -27,6 +27,10 @@ extern const field_t	dir2_leaf_entry_flds[];
 extern const field_t	dir2_leaf_hdr_flds[];
 extern const field_t	dir2_leaf_tail_flds[];
 
+extern const field_t	da_blkinfo_flds[];
+extern const field_t	da_node_entry_flds[];
+extern const field_t	da_node_hdr_flds[];
+
 /*
  * generic dir2 structures used by xfs_db
  */
db/field.c +7 -0
index 002c5ae..dc72563 100644
@@ -195,6 +195,13 @@ const ftattr_t	ftattrtab[] = {
 	  SI(bitsz(xfs_dir2_sf_off_t)), 0, NULL, NULL },
 	{ FLDT_DIR2SF, "dir2sf", NULL, (char *)dir2sf_flds, dir2sf_size,
 	  FTARG_SIZE, NULL, dir2sf_flds },
+	{ FLDT_DA_BLKINFO, "dir_blkinfo", NULL, (char *)da_blkinfo_flds,
+	  SI(bitsz(struct xfs_da_blkinfo)), 0, NULL, da_blkinfo_flds },
+	{ FLDT_DA_NODE_ENTRY, "dir_node_entry", fp_sarray,
+	  (char *)da_node_entry_flds, SI(bitsz(struct xfs_da_node_entry)), 0,
+	  NULL, da_node_entry_flds },
+	{ FLDT_DA_NODE_HDR, "dir_node_hdr", NULL, (char *)da_node_hdr_flds,
+	  SI(bitsz(struct xfs_da_node_hdr)), 0, NULL, da_node_hdr_flds },
 	{ FLDT_DIRBLOCK, "dirblock", fp_num, "%u", SI(bitsz(__uint32_t)), 0,
 	  fa_dirblock, NULL },
 	{ FLDT_DISK_DQUOT, "disk_dquot", NULL, (char *)disk_dquot_flds,
db/field.h +3 -12
index 6962d69..72c225b 100644
@@ -75,7 +75,6 @@ typedef enum fldt	{
 	FLDT_DINODE_CORE,
 	FLDT_DINODE_FMT,
 	FLDT_DINODE_U,
-	FLDT_DIR,
 	FLDT_DIR2,
 	FLDT_DIR2_BLOCK_TAIL,
 	FLDT_DIR2_DATA_FREE,
@@ -94,18 +93,10 @@ typedef enum fldt	{
 	FLDT_DIR2_SF_HDR,
 	FLDT_DIR2_SF_OFF,
 	FLDT_DIR2SF,
-	FLDT_DIR_BLKINFO,
-	FLDT_DIR_INO,
-	FLDT_DIR_LEAF_ENTRY,
-	FLDT_DIR_LEAF_HDR,
-	FLDT_DIR_LEAF_MAP,
-	FLDT_DIR_LEAF_NAME,
-	FLDT_DIR_NODE_ENTRY,
-	FLDT_DIR_NODE_HDR,
-	FLDT_DIR_SF_ENTRY,
-	FLDT_DIR_SF_HDR,
+	FLDT_DA_BLKINFO,
+	FLDT_DA_NODE_ENTRY,
+	FLDT_DA_NODE_HDR,
 	FLDT_DIRBLOCK,
-	FLDT_DIRSHORT,
 	FLDT_DISK_DQUOT,
 	FLDT_DQBLK,
 	FLDT_DQID,

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

Re: [PATCH 10/8] xfsdb: re-instate DA btree node headers

Quoted 14 lines · Dave Chinner

From: Dave Chinner<[email protected]>

When removing the dirv1 code, it wasn't immediately obvious that the dir v2 code used a small chunk of the dirv1 field definitions. i.e. those for the DA btree node headers. Hence bits of xfs_db didn't work as expected, and some tests failed in non-obvious ways. e.g test 073 failed with this additional line of output:

Use of uninitialized value $logstart in numeric gt (>) at /home/dave/src/xfstests-dev/src/fill2fs line 84

which was the result of the command:

xfs_db -r -c sb -c $TEST_DEV

Quoted 8 lines

giving an unexpectedly incorrect output.

Re-instate the needed field definitions and rename them from "DIR" to "DA" so it is obvious they are for decoding DA Btree format blocks.

---

This no longer cleanly applies.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

Re: [PATCH 10/8] xfsdb: re-instate DA btree node headers

Quoted 22 lines · Mark Tinguely
Quoted 21 lines · Dave Chinner

From: Dave Chinner<[email protected]>

When removing the dirv1 code, it wasn't immediately obvious that the dir v2 code used a small chunk of the dirv1 field definitions. i.e. those for the DA btree node headers. Hence bits of xfs_db didn't work as expected, and some tests failed in non-obvious ways. e.g test 073 failed with this additional line of output:

Use of uninitialized value $logstart in numeric gt (>) at /home/dave/src/xfstests-dev/src/fill2fs line 84

which was the result of the command:

xfs_db -r -c sb -c $TEST_DEV

giving an unexpectedly incorrect output.

Re-instate the needed field definitions and rename them from "DIR" to "DA" so it is obvious they are for decoding DA Btree format blocks.

Quoted 6 lines
Quoted 2 lines

---

This no longer cleanly applies.

--Mark.

Oops, I forgot that patch 9/8 and 10/8 was folded back into the version 2 of the tarball. 11/8 (xfs_logprint: fix continuation transactions) is not in the tarball.

Sorry for the noise.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw

[PATCH 11/8] xfs_logprint: fix continuation transactions

In reply to a message not in the archive

From: Dave Chinner <[email protected]>

As demonstrated by xfs/295, continuation transactions cause of problems for xfs_logprint. The failure demonstrated by the test is that the buffer log format structures are variable sized on disk - the dirty bitmap is sized according to the buffer length, not fixed to the length of the maximum supported buffer size.

xfs_logprint assumes that the buf log format reocrds are of fixed size, and so when a short buffer is found it fails to handle it properly and treats it like a continuation record. This causses the opheader pointer to be incremented incorrectly and then logprint wanders off into a dark corner and gets eaten by a grue.

While fixing this, make the xlog_print_record code that does the transaction opheader walking a little easier to read and stop it from outputting binary data direct to the console by converting the no-data-print case to use a hex dumping loop.

---
 logprint/log_misc.c |   22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
logprint/log_misc.c +17 -5
index 30b7ea6..d08f900 100644
@@ -268,7 +268,13 @@ xlog_print_trans_buffer(xfs_caddr_t *ptr, int len, int *i, int num_ops)
     blen = f->blf_len;
     map_size = f->blf_map_size;
     flags = f->blf_flags;
-    struct_size = sizeof(xfs_buf_log_format_t);
+
+    /*
+     * size of the format header is dependent on the size of the bitmap, not
+     * the size of the in-memory structure. Hence the slightly obtuse
+     * calculation.
+     */
+    struct_size = offsetof(struct xfs_buf_log_format, blf_map_size) + map_size;
 
     if (len >= struct_size) {
 	ASSERT((len - sizeof(struct_size)) % sizeof(int) == 0);
@@ -933,22 +939,28 @@ xlog_print_record(int			  fd,
 	continued = ((op_head->oh_flags & XLOG_WAS_CONT_TRANS) ||
 		     (op_head->oh_flags & XLOG_CONTINUE_TRANS));
 
-	/* print transaction data */
-	if (print_no_data ||
-	    (continued && be32_to_cpu(op_head->oh_len) == 0)) {
+	if (continued && be32_to_cpu(op_head->oh_len) == 0)
+		continue;
+
+	if (print_no_data) {
 	    for (n = 0; n < be32_to_cpu(op_head->oh_len); n++) {
-		printf("%c", *ptr);
+		printf("0x%02x ", (unsigned int)*ptr);
+		if (n % 16 == 15)
+			printf("\n");
 		ptr++;
 	    }
 	    printf("\n");
 	    continue;
 	}
+
+	/* print transaction data */
 	if (xlog_print_find_tid(be32_to_cpu(op_head->oh_tid),
 				op_head->oh_flags & XLOG_WAS_CONT_TRANS)) {
 	    printf(_("Left over region from split log item\n"));
 	    ptr += be32_to_cpu(op_head->oh_len);
 	    continue;
 	}
+
 	if (be32_to_cpu(op_head->oh_len) != 0) {
 	    if (*(uint *)ptr == XFS_TRANS_HEADER_MAGIC) {
 		skip = xlog_print_trans_header(&ptr,

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#7 lore raw

Re: [PATCH 11/8] xfs_logprint: fix continuation transactions

Quoted 19 lines · Dave Chinner

From: Dave Chinner<[email protected]>

As demonstrated by xfs/295, continuation transactions cause of problems for xfs_logprint. The failure demonstrated by the test is that the buffer log format structures are variable sized on disk - the dirty bitmap is sized according to the buffer length, not fixed to the length of the maximum supported buffer size.

xfs_logprint assumes that the buf log format reocrds are of fixed size, and so when a short buffer is found it fails to handle it properly and treats it like a continuation record. This causses the opheader pointer to be incremented incorrectly and then logprint wanders off into a dark corner and gets eaten by a grue.

While fixing this, make the xlog_print_record code that does the transaction opheader walking a little easier to read and stop it from outputting binary data direct to the console by converting the no-data-print case to use a hex dumping loop.

Quoted 4 lines
---
  logprint/log_misc.c |   22 +++++++++++++++++-----
  1 file changed, 17 insertions(+), 5 deletions(-)

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs