Skip to contentSkip to conversation

merge window blues, XFS broken

NOISE Michael L. Semon 3 participants 10 messages Jan 26, 2014 – Feb 12, 2014 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi! This is more an observation than a bug report, me trying to figure out what happened on what is now a 3-day-old kernel on 32-bit x86 (Pentium 4). The report is marked as [NOISE] because I can do this...

git pull origin master
git remote update # updates xfs-oss
git reset --hard v3.13
git merge xfs-oss/master

...and the resulting kernel and XFS will be as smooth as silk. However, if I do this...

git pull origin master
git remote update # at time of pull, "Already up-to-date." git merge xfs-oss/master

...the resulting XFS will not pass this, for either v4- or v5- superblock XFS:

mkfs.xfs -f $TEST_DEV      # always OK
mount $TEST_DEV $TEST_DIR  # may succeed, may fail
ls $TEST_DIR/              # may succeed, may fail
umount $TEST_DEV           # always fails

The assertion is this (from notes taken by hand):

Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Anything after my closing is supporting data. My questions are these:

1) Could the patch "xfs: format log items write directly into the linear CIL buffer" have thrown off alignment with XFS, when combined with the kernel changes for v3.14-rc?

2) Can kernel header changes--especially for this kernfs feature that keeps getting in the way--bleed into structures that XFS uses and cause chaos?

3) Because the XFS behavior changed due to some kernel change that probably isn't going away, do I need to keep worrying about this issue?

I tried to use pahole and hoped that it would point out something obvious. However, after attempting to use Perl to massage the large amount of data created by pahole, it looks like a little bit changed for XFS and a lot changed for the kernel. Lack of knowledge keeps that piece of data that must be aligned from jumping out at me.

In case I botched something in my report, I can reproduce this problem at will and can try to make a better report next time. A crash dump is available.

Thanks!

Michael

The stack trace is this (from the "crash" utility, different session):

root@plbearer:/mnt/storage/crashdump# crash vmlinux System.map vmcore
# `crash` initialization snipped

SYSTEM MAP: System.map
DEBUG KERNEL: vmlinux

    DUMPFILE: vmcore
        CPUS: 1
        DATE: Fri Jan 24 10:04:35 2014
      UPTIME: 00:02:26
LOAD AVERAGE: 0.50, 0.20, 0.08
       TASKS: 63
    NODENAME: plbearer
     RELEASE: 3.13.0+
     VERSION: #1 Fri Jan 24 09:57:19 EST 2014
     MACHINE: i686  (1794 Mhz)
      MEMORY: 1.2 GB
       PANIC: "kernel BUG at fs/xfs/xfs_message.c:107!"
         PID: 301
     COMMAND: "mount"
        TASK: c5528c30  [THREAD_INFO: bca5a000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 301 TASK: c5528c30 CPU: 0 COMMAND: "mount"
#0 [bca5bc60] crash_kexec at 7907489a
#1 [bca5bcac] do_invalid_op at 790023c8
#2 [bca5bd48] error_code (via invalid_op) at 7944b2ff

    EAX: 00000071  EBX: a40c11bc  ECX: 000002ac  EDX: c5529020  EBP: bca5bd9c 
    DS:  007b      ESI: 78135a00  ES:  007b      EDI: 78135a1c  GS:  2342
    CS:  0060      EIP: 79175065  ERR: ffffffff  EFLAGS: 00010286 

#3 [bca5bd7c] assfail at 79175065
#4 [bca5bda0] xfs_buf_item_format at 791cbd67 #5 [bca5bde8] xfs_log_commit_cil at 791cb4c8 #6 [bca5be4c] xfs_trans_commit at 7917c4fe #7 [bca5be78] xfs_log_sbcount at 79176043 #8 [bca5be8c] xfs_unmountfs at 791760f8 #9 [bca5beb4] xfs_fs_put_super at 79179257
#10 [bca5bec0] generic_shutdown_super at 790d7b5b #11 [bca5bedc] kill_block_super at 790d89be #12 [bca5beec] deactivate_locked_super at 790d7814 #13 [bca5befc] deactivate_super at 790d7871 #14 [bca5bf0c] mntput_no_expire at 790ef90b #15 [bca5bf28] mntput at 790f047f
#16 [bca5bf30] do_mount at 790f1c66
#17 [bca5bf80] sys_mount at 790f26ed
#18 [bca5bfb0] ia32_sysenter_target at 7944b5b1

    EAX: 00000015  EBX: 09bd3b70  ECX: 09bd3b80  EDX: 09bd61e0 
    DS:  007b      ESI: c0ed0000  ES:  007b      EDI: 00000000
    SS:  007b      ESP: 77c3d2f0  EBP: 00000000  GS:  0000
    CS:  0073      EIP: 6f708424  ERR: 00000015  EFLAGS: 00000246 

crash> quit

Bisect led me here:

bde7cff67c39227c6ad503394e19e58debdbc5e3 is the first bad commit commit bde7cff67c39227c6ad503394e19e58debdbc5e3

Author: Christoph Hellwig <[email protected]>
Date:   Fri Dec 13 11:34:02 2013 +1100

xfs: format log items write directly into the linear CIL buffer

# git bisect log

git bisect start
# good: [d8ec26d7f8287f5788a494f56e8814210f0e64be] Linux 3.13
git bisect good d8ec26d7f8287f5788a494f56e8814210f0e64be
# bad: [12e881fb6be0cada0ed4bebe6806945fb85f170a] nilfs2: implementation of NILFS_IOCTL_SET_SUINFO ioctl
git bisect bad 12e881fb6be0cada0ed4bebe6806945fb85f170a
# good: [de4fe30af1620b5117d65489621a5037913e7a92] Merge tag 'staging-3.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good de4fe30af1620b5117d65489621a5037913e7a92
# good: [d4371f94bc003e912d4825f5c4bdf57959857073] Merge tag 'sound-3.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect good d4371f94bc003e912d4825f5c4bdf57959857073
# good: [e1ba84597c9012b9f9075aac283ac7537d7561ba] Merge tag 'pci-v3.14-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
git bisect good e1ba84597c9012b9f9075aac283ac7537d7561ba
# good: [7ebd3faa9b5b42caf2d5aa1352a93dcfa0098011] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
git bisect good 7ebd3faa9b5b42caf2d5aa1352a93dcfa0098011
# bad: [1d32bdafaaa8bcc4c39b41ab9f674887d147f188] Merge tag 'xfs-for-linus-v3.14-rc1' of git://oss.sgi.com/xfs/xfs
git bisect bad 1d32bdafaaa8bcc4c39b41ab9f674887d147f188
# good: [93b05cba8ed52a751da9c4c7da6c97bc514bec77] Merge tag 'virtio-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good 93b05cba8ed52a751da9c4c7da6c97bc514bec77
# good: [eef334e5776c8ef547ada4cec17549929fe590b4] xfs: assert that we hold the ilock for extent map access
git bisect good eef334e5776c8ef547ada4cec17549929fe590b4
# bad: [46f23adf78545c49591619a615edeec41ed5a549] Merge branch 'xfs-factor-icluster-macros' into for-next
git bisect bad 46f23adf78545c49591619a615edeec41ed5a549
# bad: [ce8e962939ca12218092f8eb3c8cfb196cd8cc51] xfs: remove the dquot log format from the dquot log item
git bisect bad ce8e962939ca12218092f8eb3c8cfb196cd8cc51
# good: [3de559fbd04d67473b9be2bd183823c40c4b7557] xfs: refactor xfs_inode_item_format
git bisect good 3de559fbd04d67473b9be2bd183823c40c4b7557
# bad: [bde7cff67c39227c6ad503394e19e58debdbc5e3] xfs: format log items write directly into the linear CIL buffer
git bisect bad bde7cff67c39227c6ad503394e19e58debdbc5e3
# good: [1234351cba958cd5d4338172ccfc869a687cd736] xfs: introduce xlog_copy_iovec
git bisect good 1234351cba958cd5d4338172ccfc869a687cd736

# first bad commit: [bde7cff67c39227c6ad503394e19e58debdbc5e3] xfs: format log items write directly into the linear CIL buffer

Note: For the data below, I tried my hardest to run both good and bad kernels with the same kernel config. Kernel source code was generated by the git merge xfs-oss/master methods cited above.

Changes from kernel with working XFS (a) and kernel with broken XFS (b), looking for size changes:

struct xsave_struct {
a: /* size: 832, cachelines: 13, members: 3 */
b: /* size: 1088, cachelines: 17, members: 6 */
struct perf_event {
a: /* size: 864, cachelines: 14, members: 55 */
b: /* size: 872, cachelines: 14, members: 56 */
struct task_struct {
a: /* size: 2988, cachelines: 47, members: 139 */
b: /* size: 3116, cachelines: 49, members: 143 */
struct zone {
a: /* size: 764, cachelines: 12, members: 29 */
b: /* size: 768, cachelines: 12, members: 30 */
struct pglist_data {
a: /* size: 3236, cachelines: 51, members: 14 */
b: /* size: 3252, cachelines: 51, members: 14 */
struct mm_struct {
a: /* size: 564, cachelines: 9, members: 45 */
b: /* size: 576, cachelines: 9, members: 46 */
struct signal_struct {
a: /* size: 632, cachelines: 10, members: 53 */
b: /* size: 640, cachelines: 10, members: 54 */
struct xfs_dquot {
a: /* size: 516, cachelines: 9, members: 22 */
b: /* size: 492, cachelines: 8, members: 22 */
struct xfs_inode_log_item {
a: /* size: 160, cachelines: 3, members: 11 */
b: /* size: 100, cachelines: 2, members: 8 */
struct xfs_dq_logitem {
a: /* size: 104, cachelines: 2, members: 4 */
b: /* size: 80, cachelines: 2, members: 3 */
struct ftrace_event_call {
a: /* size: 72, cachelines: 2, members: 12 */
b: /* size: 76, cachelines: 2, members: 13 */
struct ftrace_event_file {
a: /* size: 36, cachelines: 1, members: 8 */
b: /* size: 48, cachelines: 1, members: 10 */
struct xfs_qoff_logitem {
a: /* size: 92, cachelines: 2, members: 3 */
b: /* size: 76, cachelines: 2, members: 3 */

Changes from kernel with working XFS (a) and kernel with broken XFS (b), looking changes in the holes in structures.

struct kernfs_open_file {
a:
b: /* XXX 3 bytes hole, try to pack */

I work with pahole maybe once every three months, and my fuzzy memory, it seems like there are more holes total on 32-bit x86 than there used to be, don't know why. As always, maybe there's a glitch in the system that created all of this. My systems evolve over time and may be in a substandard state at present.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 19 lines · Michael L. Semon

Hi! This is more an observation than a bug report, me trying to figure out what happened on what is now a 3-day-old kernel on 32-bit x86 (Pentium 4). The report is marked as [NOISE] because I can do this...

git pull origin master
git remote update # updates xfs-oss
git reset --hard v3.13
git merge xfs-oss/master

...and the resulting kernel and XFS will be as smooth as silk. However, if I do this...

git pull origin master
git remote update # at time of pull, "Already up-to-date." git merge xfs-oss/master

...the resulting XFS will not pass this, for either v4- or v5- superblock XFS:

Quoted 8 lines
mkfs.xfs -f $TEST_DEV      # always OK
mount $TEST_DEV $TEST_DIR  # may succeed, may fail
ls $TEST_DIR/              # may succeed, may fail
umount $TEST_DEV           # always fails

The assertion is this (from notes taken by hand):

Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Hmmmm - that would point to struct xfs_log_iovec being 12 bytes on ia32, similarly the struct xfs_log_vec may not be 8 byte aligned when allocated because pointers only require 4 byte alignement.

Hence the initial data buffer may be 4 byte aligned and so that will break the alignment requirement.

As a test, can you add pad both structures to be a multiple of 8 bytes in size, and add to them __aligned(8) so that they are correctly aligned in memory? i.e.

struct xfs_log_vec {
	......
	int pad;
} __aligned(8);
struct xfs_log_iovec {
	......
	int pad;
} __aligned(8);

And see if that makes the problem go away?

Quoted 3 lines

I work with pahole maybe once every three months, and my fuzzy memory, it seems like there are more holes total on 32-bit x86 than there used to be, don't know why.

Check the structs with pahole before and after you make the above padding change to see if there is a difference in size and alignment as a result.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw
Quoted 6 lines · Dave Chinner

Hmmmm - that would point to struct xfs_log_iovec being 12 bytes on ia32, similarly the struct xfs_log_vec may not be 8 byte aligned when allocated because pointers only require 4 byte alignement.

Hence the initial data buffer may be 4 byte aligned and so that will break the alignment requirement.

Yeah. Shoulkd be enough to just align the initial lv_buf allocation.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
In reply to Dave Chinner
Quoted 45 lines · Dave Chinner
Quoted 19 lines · Michael L. Semon

Hi! This is more an observation than a bug report, me trying to figure out what happened on what is now a 3-day-old kernel on 32-bit x86 (Pentium 4). The report is marked as [NOISE] because I can do this...

git pull origin master
git remote update # updates xfs-oss
git reset --hard v3.13
git merge xfs-oss/master

...and the resulting kernel and XFS will be as smooth as silk. However, if I do this...

git pull origin master
git remote update # at time of pull, "Already up-to-date." git merge xfs-oss/master

...the resulting XFS will not pass this, for either v4- or v5- superblock XFS:

Quoted 8 lines
mkfs.xfs -f $TEST_DEV      # always OK
mount $TEST_DEV $TEST_DIR  # may succeed, may fail
ls $TEST_DIR/              # may succeed, may fail
umount $TEST_DEV           # always fails

The assertion is this (from notes taken by hand):

Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Hmmmm - that would point to struct xfs_log_iovec being 12 bytes on ia32, similarly the struct xfs_log_vec may not be 8 byte aligned when allocated because pointers only require 4 byte alignement.

Hence the initial data buffer may be 4 byte aligned and so that will break the alignment requirement.

As a test, can you add pad both structures to be a multiple of 8 bytes in size, and add to them __aligned(8) so that they are correctly aligned in memory? i.e.

struct xfs_log_vec {
	......
	int pad;
} __aligned(8);
Quoted 6 lines
struct xfs_log_iovec {
	......
	int pad;
} __aligned(8);

And see if that makes the problem go away?

No. However, I didn't try to pad anywhere except at the end of each struct. Also, I didn't know how to handle the mix of __aligned(8) within a typedef definition.

What I did do however, is add a similar pad to xfs_inode_log_item, and that got both umounts and xfstests working again. There's a new lockdep that I'll have to submit tomorrow, but otherwise it looks consistent with previous XFS behavior in very, very little testing.

Suggestions are welcome. If there was a list of what is supposed to be 8-byte aligned, I'd be happy to look further.

Quoted 11 lines
Quoted 3 lines

I work with pahole maybe once every three months, and my fuzzy memory, it seems like there are more holes total on 32-bit x86 than there used to be, don't know why.

Check the structs with pahole before and after you make the above padding change to see if there is a difference in size and alignment as a result.

Cheers,

Dave.

Everything after my closing is just supporting data, and a verification that last night's git pull didn't have a non-XFS fix to make the problem go away. Just grep "pahole reports:" to get to the pahole results.

Thanks, Dave and Christoph!

Michael

# ======================= MERGE ========================

# display kernel commit
root@plbearer:/usr/src/kernel-git/linux# git log | head -n 8
commit b2e448eca1a52fea181905845728ae00a138d84e
Merge: 2d2e7d1 d02b370
Author: Linus Torvalds <[email protected]>
Date:   Sat Jan 25 15:33:41 2014 -0800

Merge branch 'ipmi' (ipmi patches from Corey Minyard)

Merge ipmi fixes from Corey Minyard:

# merge
root@plbearer:/usr/src/kernel-git/linux# git merge xfs-oss/master

Already up-to-date.

root@plbearer:/usr/src/kernel-git/linux# zcat /proc/config.gz > .config
root@plbearer:/usr/src/kernel-git/linux# make oldconfig
# and so on and so forth...

# ============================== BEFORE ==============================

# pahole reports:
struct xfs_log_iovec {
	void *                     i_addr;               /*     0     4 */
	int                        i_len;                /*     4     4 */
	uint                       i_type;               /*     8     4 */
	/* size: 12, cachelines: 1, members: 3 */
	/* last cacheline: 12 bytes */
};

struct xfs_inode_log_item {
xfs_log_item_t ili_item; /* 0 68 */
/* --- cacheline 1 boundary (64 bytes) was 4 bytes ago --- */
struct xfs_inode * ili_inode; /* 68 4 */
xfs_lsn_t ili_flush_lsn; /* 72 8 */
xfs_lsn_t ili_last_lsn; /* 80 8 */
short unsigned int ili_lock_flags; /* 88 2 */
short unsigned int ili_logged; /* 90 2 */
unsigned int ili_last_fields; /* 92 4 */
unsigned int ili_fields; /* 96 4 */

/* size: 100, cachelines: 2, members: 8 */ /* last cacheline: 36 bytes */
};
struct xfs_log_vec {

	struct xfs_log_vec *       lv_next;              /*     0     4 */
	int                        lv_niovecs;           /*     4     4 */
	struct xfs_log_iovec *     lv_iovecp;            /*     8     4 */
	struct xfs_log_item *      lv_item;              /*    12     4 */
	char *                     lv_buf;               /*    16     4 */
	int                        lv_buf_len;           /*    20     4 */
	int                        lv_size;              /*    24     4 */
	/* size: 28, cachelines: 1, members: 7 */
	/* last cacheline: 28 bytes */
};
# ============== CHECK TO SEE IF KERNEL STILL SHOWS ISSUE =============
# serial login:
mls@oldsvrhw:~$ cu -l /dev/ttyS0 -s 115200
Connected.
root
Password: 
Last login: Sun Jan 26 23:09:13 -0500 2014 on /dev/tty4.
root@plbearer:~# dmesg -c > /dev/null
root@plbearer:~# dmesg --console-on
root@plbearer:~# dmesg --console-level debug
root@plbearer:~# mkfs.xfs -f $TEST_DEV
meta-data=/dev/md3p3             isize=256    agcount=8, agsize=131056 blks
         =                       sectsz=512   attr=2, projid32bit=1, parent=0
         =                       crc=0
data     =                       bsize=4096   blocks=1048448, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=12800, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@plbearer:~# mount $TEST_DEV $TEST_DIR
[   87.438116] XFS (md3p3): Mounting Filesystem
[   87.630320] XFS (md3p3): Ending clean mount
root@plbearer:~# ls $TEST_DIR/
[   94.140207] XFS: Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Entering kdb (current=0xc5298c30, pid 297) Oops: (null) due to oops @ 0x791752c5

CPU: 0 PID: 297 Comm: ls Not tainted 3.13.0+ #1
Hardware name: Dell Computer Corporation Dimension 2350/07W080, BIOS A01 12/17/2002
task: c5298c30 ti: c520e000 task.ti: c520e000
EIP: 0060:[<791752c5>] EFLAGS: 00010286 CPU: 0
EIP is at assfail+0x2b/0x2d
EAX: 00000071 EBX: c60ba600 ECX: 00000296 EDX: c5299098
ESI: c60ba61c EDI: c60ba600 EBP: c520fe40 ESP: c520fe2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
CR0: 80050033 CR2: 08f1612c CR3: 4d1f0000 CR4: 000007d0
Stack:
 00000000 79570bc8 79576e28 7956946d 00000031 c520fe70 791ce45f c520fe70
 7917ceb0 c520fec4 c50d5068 c520fe70 c55d8000 00000000 c50d5068 c607ae30
 c60ba600 c520fed4 791cb72b c607af80 c6c01e80 000000d8 c4294000 c520feec
Call Trace:
 [<791ce45f>] xfs_inode_item_format+0x4a/0x1c5
 [<7917ceb0>] ? kmem_alloc+0x64/0xdf
 [<791cb72b>] xfs_log_commit_cil+0x391/0x4c4
 [<7917c763>] xfs_trans_commit+0xac/0x230
 [<79172cf1>] xfs_vn_update_time+0xdb/0x142
 [<79172c16>] ? xfs_setattr_mode.isra.10+0x63/0x63
 [<790eb7f2>] update_time+0x1e/0x9e
 [<790ed28c>] touch_atime+0xcb/0x103
 [<790e5e89>] iterate_dir+0x8f/0x9b
 [<790e6041>] SyS_getdents64+0x6d/0xcc
 [<790e5d18>] ? filldir+0xc7/0xc7
 [<7944e938>] sysenter_do_call+0x12/0x36
Code: 
55 89 e5 83 ec 14 3e 8d 74 26 00  89  4c 24 10 89
54 24 0c 89 44 24 08 c7 44 24 04  c8  0b 57 79 c7 

04 24 00 00 00 00 e8 ad fd ff ff <0f> 0b 55 89 e5
83 ec 14 3e 8d 74 26 00 c7 44 24 10 01 00 00 00

# ============================== PATCH ============================== # I present this only so you can point out the error of my ways or

# try this yourself.  God help the poor soul who finds this through 
# Google and applies it.
Quoted 1 line

From 7e92813556e5cf3fe466680be268efa9dd7e8776 Mon Sep 17 00:00:00 2001

From: "Michael L. Semon" <[email protected]>
Date: Mon, 27 Jan 2014 02:33:04 -0500
Subject: [PATCH] xfs: add alignment aids to three log-related structs

A change in the kernel after 3.13 altered the alignment of several structures, causing all XFS umounts to fail. xfs_inode_log_item was padded to fix this. Along the way, xfs_log_vec and xfs_log_iovec were also padded to aid in solving the problem.

---
 fs/xfs/xfs_inode_item.h | 7 +++++--
 fs/xfs/xfs_log.h        | 3 ++-
 fs/xfs/xfs_log_format.h | 7 +++++--
 3 files changed, 12 insertions(+), 5 deletions(-)
fs/xfs/xfs_inode_item.h +5 -2
index 488d812..820aac8 100644
@@ -25,7 +25,7 @@ struct xfs_bmbt_rec;
 struct xfs_inode;
 struct xfs_mount;
 
-typedef struct xfs_inode_log_item {
+struct xfs_inode_log_item {
 	xfs_log_item_t		ili_item;	   /* common portion */
 	struct xfs_inode	*ili_inode;	   /* inode ptr */
 	xfs_lsn_t		ili_flush_lsn;	   /* lsn at last flush */
@@ -34,7 +34,10 @@ typedef struct xfs_inode_log_item {
 	unsigned short		ili_logged;	   /* flushed logged data */
 	unsigned int		ili_last_fields;   /* fields when flushed */
 	unsigned int		ili_fields;	   /* fields to be logged */
-} xfs_inode_log_item_t;
+	int			pad;		   /* for 8-byte alignment */
+} __aligned(8);
+
+typedef struct xfs_inode_log_item xfs_inode_log_item_t;
 
 static inline int xfs_inode_clean(xfs_inode_t *ip)
 {
fs/xfs/xfs_log.h +2 -1
index b0f4ef7..488833c 100644
@@ -26,7 +26,8 @@ struct xfs_log_vec {
 	char			*lv_buf;	/* formatted buffer */
 	int			lv_buf_len;	/* size of formatted buffer */
 	int			lv_size;	/* size of allocated lv */
-};
+	int			pad;		/* for 8-byte alignment */
+} __aligned(8);
 
 #define XFS_LOG_VEC_ORDERED	(-1)
 
fs/xfs/xfs_log_format.h +5 -2
index f0969c7..d3174cb 100644
@@ -184,11 +184,14 @@ typedef union xlog_in_core2 {
 } xlog_in_core_2_t;
 
 /* not an on-disk structure, but needed by log recovery in userspace */
-typedef struct xfs_log_iovec {
+struct xfs_log_iovec {
 	void		*i_addr;	/* beginning address of region */
 	int		i_len;		/* length in bytes of region */
 	uint		i_type;		/* type of region */
-} xfs_log_iovec_t;
+	int		pad;		/* for 8-byte alignment */
+} __aligned(8);
+
+typedef struct xfs_log_iovec xfs_log_iovec_t;
 
 
 /*
Signature · 46 lines
1.8.4

# ============================== AFTER ===============================
# pahole reports:
struct xfs_log_iovec {
	void *                     i_addr;               /*     0     4 */
	int                        i_len;                /*     4     4 */
	uint                       i_type;               /*     8     4 */
	int                        pad;                  /*    12     4 */

	/* size: 16, cachelines: 1, members: 4 */
	/* last cacheline: 16 bytes */
};
struct xfs_inode_log_item {
	xfs_log_item_t             ili_item;             /*     0    68 */
	/* --- cacheline 1 boundary (64 bytes) was 4 bytes ago --- */
	struct xfs_inode *         ili_inode;            /*    68     4 */
	xfs_lsn_t                  ili_flush_lsn;        /*    72     8 */
	xfs_lsn_t                  ili_last_lsn;         /*    80     8 */
	short unsigned int         ili_lock_flags;       /*    88     2 */
	short unsigned int         ili_logged;           /*    90     2 */
	unsigned int               ili_last_fields;      /*    92     4 */
	unsigned int               ili_fields;           /*    96     4 */
	int                        pad;                  /*   100     4 */

	/* size: 104, cachelines: 2, members: 9 */
	/* last cacheline: 40 bytes */
};
struct xfs_log_vec {
	struct xfs_log_vec *       lv_next;              /*     0     4 */
	int                        lv_niovecs;           /*     4     4 */
	struct xfs_log_iovec *     lv_iovecp;            /*     8     4 */
	struct xfs_log_item *      lv_item;              /*    12     4 */
	char *                     lv_buf;               /*    16     4 */
	int                        lv_buf_len;           /*    20     4 */
	int                        lv_size;              /*    24     4 */
	int                        pad;                  /*    28     4 */

	/* size: 32, cachelines: 1, members: 8 */
	/* last cacheline: 32 bytes */
};

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 6 lines · Michael L. Semon
root@plbearer:~# ls $TEST_DIR/
[   94.140207] XFS: Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Entering kdb (current=0xc5298c30, pid 297) Oops: (null) due to oops @ 0x791752c5

Quoted 15 lines
CPU: 0 PID: 297 Comm: ls Not tainted 3.13.0+ #1
Hardware name: Dell Computer Corporation Dimension 2350/07W080, BIOS A01 12/17/2002
task: c5298c30 ti: c520e000 task.ti: c520e000
EIP: 0060:[<791752c5>] EFLAGS: 00010286 CPU: 0
EIP is at assfail+0x2b/0x2d
EAX: 00000071 EBX: c60ba600 ECX: 00000296 EDX: c5299098
ESI: c60ba61c EDI: c60ba600 EBP: c520fe40 ESP: c520fe2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
CR0: 80050033 CR2: 08f1612c CR3: 4d1f0000 CR4: 000007d0
Stack:
 00000000 79570bc8 79576e28 7956946d 00000031 c520fe70 791ce45f c520fe70
 7917ceb0 c520fec4 c50d5068 c520fe70 c55d8000 00000000 c50d5068 c607ae30
 c60ba600 c520fed4 791cb72b c607af80 c6c01e80 000000d8 c4294000 c520feec
Call Trace:
 [<791ce45f>] xfs_inode_item_format+0x4a/0x1c5

It's not clear to me that there's anything wrong with the inode log item structure, so I need to know what iovec we tripped over here. Can you post the disassembly of this function so we can see which call to xlog_prepare_iovec tripped the assert? i.e.:

gdb> disass xfs_inode_item_format

to give the raw disassembly output, and

gdb> disass /m xfs_inode_item_format

To output the c-code annotated version.

Or even just annotating the code with printk()s prior to each xlog_prepare_iovec() call in xfs_inode_item_format will do ;)

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
Quoted 34 lines · Dave Chinner
Quoted 6 lines · Michael L. Semon
root@plbearer:~# ls $TEST_DIR/
[   94.140207] XFS: Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Entering kdb (current=0xc5298c30, pid 297) Oops: (null) due to oops @ 0x791752c5

Quoted 15 lines
CPU: 0 PID: 297 Comm: ls Not tainted 3.13.0+ #1
Hardware name: Dell Computer Corporation Dimension 2350/07W080, BIOS A01 12/17/2002
task: c5298c30 ti: c520e000 task.ti: c520e000
EIP: 0060:[<791752c5>] EFLAGS: 00010286 CPU: 0
EIP is at assfail+0x2b/0x2d
EAX: 00000071 EBX: c60ba600 ECX: 00000296 EDX: c5299098
ESI: c60ba61c EDI: c60ba600 EBP: c520fe40 ESP: c520fe2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
CR0: 80050033 CR2: 08f1612c CR3: 4d1f0000 CR4: 000007d0
Stack:
 00000000 79570bc8 79576e28 7956946d 00000031 c520fe70 791ce45f c520fe70
 7917ceb0 c520fec4 c50d5068 c520fe70 c55d8000 00000000 c50d5068 c607ae30
 c60ba600 c520fed4 791cb72b c607af80 c6c01e80 000000d8 c4294000 c520feec
Call Trace:
 [<791ce45f>] xfs_inode_item_format+0x4a/0x1c5

It's not clear to me that there's anything wrong with the inode log item structure, so I need to know what iovec we tripped over here. Can you post the disassembly of this function so we can see which call to xlog_prepare_iovec tripped the assert? i.e.:

gdb> disass xfs_inode_item_format

to give the raw disassembly output, and

gdb> disass /m xfs_inode_item_format

Quoted 9 lines

To output the c-code annotated version.

Or even just annotating the code with printk()s prior to each xlog_prepare_iovec() call in xfs_inode_item_format will do ;)

Cheers,

Dave.

OK, I had to generate a new crash for this, so pardon the dust:

# ======= SERIAL SESSION

root@plbearer:/var/lib/xfstests# mkfs.xfs -f $TEST_DEV
meta-data=/dev/md3p3             isize=256    agcount=8, agsize=131056 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=1048448, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=12800, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@plbearer:/var/lib/xfstests# mount $TEST_DEV $TEST_DIR
root@plbearer:/var/lib/xfstests# ls $TEST_DIR/
[  218.561794] XFS: Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Entering kdb (current=0xc5289860, pid 320) Oops: (null) due to oops @ 0x791752c5

CPU: 0 PID: 320 Comm: ls Not tainted 3.13.0+ #12
Hardware name: Dell Computer Corporation Dimension 2350/07W080, BIOS A01 12/17/2002
task: c5289860 ti: c5210000 task.ti: c5210000
EIP: 0060:[<791752c5>] EFLAGS: 00010286 CPU: 0
EIP is at assfail+0x2b/0x2d
EAX: 00000071 EBX: c5095500 ECX: 000002a3 EDX: c5289cc8
ESI: c509551c EDI: c5095500 EBP: c5211e40 ESP: c5211e2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
CR0: 80050033 CR2: 0860f12c CR3: 4c2a8000 CR4: 000007d0
Stack:
 00000000 79571d00 79577f60 7956a5a5 00000031 c5211e70 791ce45f c5211e70
 7917ceb0 c5211ec4 c6356068 c5211e70 c55cc000 00000000 c6356068 c607ab10
 c5095500 c5211ed4 791cb72b c607af80 c6c01e80 000000d8 c5121000 c5211eec
Call Trace:
 [<791ce45f>] xfs_inode_item_format+0x4a/0x1c5
 [<7917ceb0>] ? kmem_alloc+0x64/0xdf
 [<791cb72b>] xfs_log_commit_cil+0x391/0x4c4
 [<7917c763>] xfs_trans_commit+0xac/0x230
 [<79172cf1>] xfs_vn_update_time+0xdb/0x142
 [<79172c16>] ? xfs_setattr_mode.isra.10+0x63/0x63
 [<790eb7f2>] update_time+0x1e/0x9e
 [<790ed28c>] touch_atime+0xcb/0x103
 [<790e5e89>] iterate_dir+0x8f/0x9b
 [<790e6041>] SyS_getdents64+0x6d/0xcc
 [<790e5d18>] ? filldir+0xc7/0xc7
 [<7944f1b8>] sysenter_do_call+0x12/0x36
Code: 
55 89 e5 83 ec 14 3e 8d 74 26 00  89  4c 24 10 89 
54 24 0c 89 44 24 08 c7 44 24 04  00  1d 57 79 c7 

04 24 00 00 00 00 e8 ad fd ff ff <0f> 0b 55 89 e5 83 ec 14 3e 8d 74 26 00 c7 44 24 10 01 00 00 00

# ===== CRASH SESSION

root@plbearer:/mnt/storage/crashdump# crash vmlinux System.map vmcore
# setup was snipped

SYSTEM MAP: System.map
DEBUG KERNEL: vmlinux

    DUMPFILE: vmcore
        CPUS: 1
        DATE: Mon Jan 27 23:39:03 2014
      UPTIME: 00:03:38
LOAD AVERAGE: 0.06, 0.04, 0.02
       TASKS: 63
    NODENAME: plbearer
     RELEASE: 3.13.0+
     VERSION: #12 Mon Jan 27 23:30:59 EST 2014
     MACHINE: i686  (1794 Mhz)
      MEMORY: 1.2 GB
       PANIC: "kernel BUG at fs/xfs/xfs_message.c:107!"
         PID: 320
     COMMAND: "ls"
        TASK: c5289860  [THREAD_INFO: c5210000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 320 TASK: c5289860 CPU: 0 COMMAND: "ls"
#0 [c5211d04] crash_kexec at 79074890
#1 [c5211d50] do_invalid_op at 790023c8
#2 [c5211dec] error_code (via invalid_op) at 7944eeef

    EAX: 00000071  EBX: c5095500  ECX: 000002a3  EDX: c5289cc8  EBP: c5211e40 
    DS:  007b      ESI: c509551c  ES:  007b      EDI: c5095500  GS:  2342
    CS:  0060      EIP: 791752c5  ERR: ffffffff  EFLAGS: 00010286 

#3 [c5211e20] assfail at 791752c5
#4 [c5211e44] xfs_inode_item_format at 791ce45a #5 [c5211e74] xfs_log_commit_cil at 791cb728 #6 [c5211ed8] xfs_trans_commit at 7917c75e
#7 [c5211f04] xfs_vn_update_time at 79172cec #8 [c5211f28] update_time at 790eb7f0
#9 [c5211f44] touch_atime at 790ed287
#10 [c5211f5c] iterate_dir at 790e5e84
#11 [c5211f78] sys_getdents64 at 790e603c
#12 [c5211fb0] ia32_sysenter_target at 7944f1b1

    EAX: 000000dc  EBX: 00000003  ECX: 08607128  EDX: 00008000 
    DS:  007b      ESI: 08607128  ES:  007b      EDI: 6f790000
    SS:  007b      ESP: 778278b0  EBP: 00000000  GS:  0000
    CS:  0073      EIP: 6f7b2424  ERR: 000000dc  EFLAGS: 00000216 
crash> gdb disass /m xfs_inode_item_format
Dump of assembler code for function xfs_inode_item_format:
367     {
   0x791ce415 <+0>:     push   %ebp
   0x791ce416 <+1>:     mov    %esp,%ebp
   0x791ce418 <+3>:     push   %edi
   0x791ce419 <+4>:     push   %esi
   0x791ce41a <+5>:     push   %ebx
   0x791ce41b <+6>:     sub    $0x1c,%esp
   0x791ce41e <+9>:     lea    %ds:0x0(%esi,%eiz,1),%esi
   0x791ce423 <+14>:    mov    %eax,-0x1c(%ebp)
   0x791ce426 <+17>:    mov    %edx,%ebx

368 struct xfs_inode_log_item *iip = INODE_ITEM(lip);
369 struct xfs_inode *ip = iip->ili_inode;

   0x791ce428 <+19>:    mov    0x44(%eax),%eax
   0x791ce42b <+22>:    mov    %eax,-0x14(%ebp)

370 struct xfs_inode_log_format *ilf;
371 struct xfs_log_iovec *vecp = NULL;
0x791ce42e <+25>: movl $0x0,-0x10(%ebp)

372
373 ilf = xlog_prepare_iovec(lv, &vecp, XLOG_REG_TYPE_IFORMAT);
374 ilf->ilf_type = XFS_LI_INODE;
0x791ce464 <+79>: movw $0x123b,(%esi)

375 ilf->ilf_ino = ip->i_ino;

   0x791ce469 <+84>:    mov    -0x14(%ebp),%ecx
   0x791ce46c <+87>:    mov    0x14(%ecx),%edx
   0x791ce46f <+90>:    mov    0x10(%ecx),%eax
   0x791ce472 <+93>:    mov    %eax,0xc(%esi)
   0x791ce475 <+96>:    mov    %edx,0x10(%esi)

376 ilf->ilf_blkno = ip->i_imap.im_blkno;

   0x791ce478 <+99>:    mov    0x1c(%ecx),%edx
   0x791ce47b <+102>:   mov    0x18(%ecx),%eax
   0x791ce47e <+105>:   mov    %eax,0x24(%esi)
   0x791ce481 <+108>:   mov    %edx,0x28(%esi)

377 ilf->ilf_len = ip->i_imap.im_len;
0x791ce484 <+111>: movzwl 0x20(%ecx),%eax
0x791ce488 <+115>: mov %eax,0x2c(%esi)

378 ilf->ilf_boffset = ip->i_imap.im_boffset;

   0x791ce48b <+118>:   mov    %ecx,%edx
   0x791ce48d <+120>:   movzwl 0x22(%ecx),%eax
   0x791ce491 <+124>:   mov    %eax,0x30(%esi)

379 ilf->ilf_fields = XFS_ILOG_CORE;
0x791ce494 <+127>: movl $0x1,0x4(%esi)

380 ilf->ilf_size = 2; /* format + core */
0x791ce49b <+134>: movw $0x2,0x2(%esi)

381 xlog_finish_iovec(lv, vecp, sizeof(struct xfs_inode_log_format));
0x791ce4a1 <+140>: mov -0x10(%ebp),%edi

382
383 if (ip->i_d.di_version == 1)

   0x791ce4af <+154>:   mov    %ecx,%eax
   0x791ce4b1 <+156>:   movzbl 0x13c(%ecx),%ecx
   0x791ce4b8 <+163>:   cmp    $0x1,%cl
   0x791ce4bb <+166>:   je     0x791ce5c1 <xfs_inode_item_format+428>
   0x791ce4c1 <+172>:   mov    %cl,-0x18(%ebp)
384                     xfs_inode_item_format_v1_inode(ip);
   0x791ce5c1 <+428>:   mov    %edx,%edi
   0x791ce5c3 <+430>:   call   0x791cdc1e <xfs_inode_item_format_v1_inode>
   0x791ce5c8 <+435>:   movzbl 0x13c(%edi),%eax
   0x791ce5cf <+442>:   mov    %al,-0x18(%ebp)
   0x791ce5d2 <+445>:   mov    -0x10(%ebp),%edi
   0x791ce5d5 <+448>:   jmp    0x791ce4c4 <xfs_inode_item_format+175>

385 xlog_copy_iovec(lv, &vecp, XLOG_REG_TYPE_ICORE,
386 &ip->i_d,

   0x791ce4dd <+200>:   mov    -0x14(%ebp),%eax
   0x791ce4e0 <+203>:   add    $0x138,%eax
   0x791ce4e5 <+208>:   mov    %eax,-0x18(%ebp)

387 xfs_icdinode_size(ip->i_d.di_version));
388
389 xfs_inode_item_format_data_fork(iip, ilf, lv, &vecp);

   0x791ce55d <+328>:   lea    -0x10(%ebp),%edi
   0x791ce560 <+331>:   mov    %edi,(%esp)
   0x791ce563 <+334>:   mov    %ebx,%ecx
   0x791ce565 <+336>:   mov    %esi,%edx
   0x791ce567 <+338>:   mov    -0x1c(%ebp),%eax
   0x791ce56a <+341>:   call   0x791cdc99 <xfs_inode_item_format_data_fork>

390 if (XFS_IFORK_Q(ip)) {

   0x791ce56f <+346>:   mov    -0x14(%ebp),%eax
   0x791ce572 <+349>:   cmpb   $0x0,0x18a(%eax)
   0x791ce579 <+356>:   jne    0x791ce597 <xfs_inode_item_format+386>
391                     xfs_inode_item_format_attr_fork(iip, ilf, lv, &vecp);
   0x791ce597 <+386>:   mov    %edi,(%esp)
   0x791ce59a <+389>:   mov    %ebx,%ecx
   0x791ce59c <+391>:   mov    %esi,%edx
   0x791ce59e <+393>:   mov    -0x1c(%ebp),%ebx
   0x791ce5a1 <+396>:   mov    %ebx,%eax
   0x791ce5a3 <+398>:   call   0x791ce09d <xfs_inode_item_format_attr_fork>
   0x791ce5a8 <+403>:   mov    0x60(%ebx),%eax

392 } else {

393                     iip->ili_fields &=
   0x791ce57b <+358>:   mov    -0x1c(%ebp),%ebx
   0x791ce57e <+361>:   mov    0x60(%ebx),%eax
   0x791ce581 <+364>:   and    $0xfffffe3f,%eax
   0x791ce586 <+369>:   mov    %eax,0x60(%ebx)

394 ~(XFS_ILOG_ADATA | XFS_ILOG_ABROOT | XFS_ILOG_AEXT);
395 }
396
397 /* update the format with the exact fields we actually logged */
398 ilf->ilf_fields |= (iip->ili_fields & ~XFS_ILOG_TIMESTAMP);

   0x791ce589 <+372>:   and    $0xbf,%ah
   0x791ce58c <+375>:   or     %eax,0x4(%esi)
   0x791ce5ab <+406>:   and    $0xbf,%ah
   0x791ce5ae <+409>:   or     %eax,0x4(%esi)

399 }

   0x791ce58f <+378>:   add    $0x1c,%esp
   0x791ce592 <+381>:   pop    %ebx
   0x791ce593 <+382>:   pop    %esi
   0x791ce594 <+383>:   pop    %edi
   0x791ce595 <+384>:   pop    %ebp
   0x791ce596 <+385>:   ret    
   0x791ce5b1 <+412>:   add    $0x1c,%esp
   0x791ce5b4 <+415>:   pop    %ebx
   0x791ce5b5 <+416>:   pop    %esi
   0x791ce5b6 <+417>:   pop    %edi
   0x791ce5b7 <+418>:   pop    %ebp
   0x791ce5b8 <+419>:   ret    

End of assembler dump.
crash> quit

root@plbearer:/mnt/storage/crashdump# exit

Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
Quoted 28 lines · Michael L. Semon
Quoted 27 lines · Dave Chinner
Quoted 6 lines · Michael L. Semon
root@plbearer:~# ls $TEST_DIR/
[   94.140207] XFS: Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Entering kdb (current=0xc5298c30, pid 297) Oops: (null) due to oops @ 0x791752c5

Quoted 15 lines
CPU: 0 PID: 297 Comm: ls Not tainted 3.13.0+ #1
Hardware name: Dell Computer Corporation Dimension 2350/07W080, BIOS A01 12/17/2002
task: c5298c30 ti: c520e000 task.ti: c520e000
EIP: 0060:[<791752c5>] EFLAGS: 00010286 CPU: 0
EIP is at assfail+0x2b/0x2d
EAX: 00000071 EBX: c60ba600 ECX: 00000296 EDX: c5299098
ESI: c60ba61c EDI: c60ba600 EBP: c520fe40 ESP: c520fe2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
CR0: 80050033 CR2: 08f1612c CR3: 4d1f0000 CR4: 000007d0
Stack:
 00000000 79570bc8 79576e28 7956946d 00000031 c520fe70 791ce45f c520fe70
 7917ceb0 c520fec4 c50d5068 c520fe70 c55d8000 00000000 c50d5068 c607ae30
 c60ba600 c520fed4 791cb72b c607af80 c6c01e80 000000d8 c4294000 c520feec
Call Trace:
 [<791ce45f>] xfs_inode_item_format+0x4a/0x1c5

It's not clear to me that there's anything wrong with the inode log item structure, so I need to know what iovec we tripped over here. Can you post the disassembly of this function so we can see which call to xlog_prepare_iovec tripped the assert? i.e.:

.....

Quoted 1 line

OK, I had to generate a new crash for this, so pardon the dust:

Actually, you don't ned to crash a kernel to do this. Just run 'gdb vmlinux' from your build directory....

Quoted 18 lines

crash> gdb disass /m xfs_inode_item_format
Dump of assembler code for function xfs_inode_item_format:

367     {
   0x791ce415 <+0>:     push   %ebp
   0x791ce416 <+1>:     mov    %esp,%ebp
   0x791ce418 <+3>:     push   %edi
   0x791ce419 <+4>:     push   %esi
   0x791ce41a <+5>:     push   %ebx
   0x791ce41b <+6>:     sub    $0x1c,%esp
   0x791ce41e <+9>:     lea    %ds:0x0(%esi,%eiz,1),%esi
   0x791ce423 <+14>:    mov    %eax,-0x1c(%ebp)
   0x791ce426 <+17>:    mov    %edx,%ebx

368 struct xfs_inode_log_item *iip = INODE_ITEM(lip);

369             struct xfs_inode        *ip = iip->ili_inode;
   0x791ce428 <+19>:    mov    0x44(%eax),%eax
   0x791ce42b <+22>:    mov    %eax,-0x14(%ebp)
Quoted 8 lines

370 struct xfs_inode_log_format *ilf;

371             struct xfs_log_iovec    *vecp = NULL;
   0x791ce42e <+25>:    movl   $0x0,-0x10(%ebp)

372
373 ilf = xlog_prepare_iovec(lv, &vecp, XLOG_REG_TYPE_IFORMAT);

374             ilf->ilf_type = XFS_LI_INODE;
   0x791ce464 <+79>:    movw   $0x123b,(%esi)

Ok, so xfs_inode_item_format+0x4a is inside the very first call to preapre the ilf structure. That tells us that the initial xfs_log_vec/xfs_log_iovec array are resulting in an unaligned buffer.

Can you try the patch below, Michael?

Cheers,

Dave.

Signature · 20 lines
Dave Chinner
[email protected]

xfs: ensure correct log item buffer alignment

From: Dave Chinner <[email protected]>

On 32 bit platforms, the log item vector headers are not 64 bit
aligned or sized. hence if we don't take care to align them
correctly or pad the buffer appropriately for 8 byte alignment, we
can end up with alignment issues when accessing the user buffer
directly as a structure.

To solve this, simply pad the buffer headers to 64 bit offset so
that the data section is always 8 byte aligned. 

Signed-off-by: Dave Chinner <[email protected]>
---
 fs/xfs/xfs_log_cil.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
fs/xfs/xfs_log_cil.c +15 -4
index cdebd83..4ef6fdb 100644
@@ -205,16 +205,25 @@ xlog_cil_insert_format_items(
 		/*
 		 * We 64-bit align the length of each iovec so that the start
 		 * of the next one is naturally aligned.  We'll need to
-		 * account for that slack space here.
+		 * account for that slack space here. Then round nbytes up
+		 * to 64-bit alignment so that the initial buffer alignment is
+		 * easy to calculate and verify.
 		 */
 		nbytes += niovecs * sizeof(uint64_t);
+		nbytes = round_up(nbytes, sizeof(uint64_t));
 
 		/* grab the old item if it exists for reservation accounting */
 		old_lv = lip->li_lv;
 
-		/* calc buffer size */
-		buf_size = sizeof(struct xfs_log_vec) + nbytes +
-				niovecs * sizeof(struct xfs_log_iovec);
+		/*
+		 * The data buffer needs to start 64-bit aligned, so round up
+		 * that space to ensure we can align it appropriately and not
+		 * overrun the buffer.
+		 */
+		buf_size = nbytes +
+			   round_up((sizeof(struct xfs_log_vec) +
+				     niovecs * sizeof(struct xfs_log_iovec)),
+				    sizeof(uint64_t));
 
 		/* compare to existing item size */
 		if (lip->li_lv && buf_size <= lip->li_lv->lv_size) {
@@ -251,6 +260,8 @@ xlog_cil_insert_format_items(
 		/* The allocated data region lies beyond the iovec region */
 		lv->lv_buf_len = 0;
 		lv->lv_buf = (char *)lv + buf_size - nbytes;
+		ASSERT(IS_ALIGNED((unsigned long)lv->lv_buf, sizeof(uint64_t)));
+
 		lip->li_ops->iop_format(lip, lv);
 insert:
 		ASSERT(lv->lv_buf_len <= nbytes);

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
Quoted 10 lines · Dave Chinner

Ok, so xfs_inode_item_format+0x4a is inside the very first call to preapre the ilf structure. That tells us that the initial xfs_log_vec/xfs_log_iovec array are resulting in an unaligned buffer.

Can you try the patch below, Michael?

Cheers,

Dave.

To the best of my knowledge, it works fine. It was subjected to some of the xfstests xfs/* and generic/* series along with some of the "log" group. In addition, it was run through fs_mark and a homebrew benchmark idea of running a ( make clean; make ) loop for the kernel while two fsx processes were running. This was for default (4k, internal logdev) v5-superblock XFS filesystems. v4-superblock XFS went through similar testing, to a lesser degree.

I did not know about a message like "log buf needs to be larger than stripe size" on tests that require v2 logs. [xfs/087 might be one of those tests.] This was also the case for the test-patch that padded the structs directly, and this might be an old message.

At the very least, I'm keeping this patch. Your approach to the solution looks cool. Basically, you're stating that you can have either xfs_log_vec or xfs_log_iovec in there, and no matter what their size, they get aligned in the log, correct?

Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
In reply to Dave Chinner
Quoted 62 lines · Dave Chinner
Quoted 28 lines · Michael L. Semon
Quoted 27 lines · Dave Chinner
Quoted 6 lines · Michael L. Semon
root@plbearer:~# ls $TEST_DIR/
[   94.140207] XFS: Assertion failed: IS_ALIGNED((unsigned long)vec->i_addr, sizeof(uint64_t)), file: fs/xfs/xfs_log.h, line: 49

Entering kdb (current=0xc5298c30, pid 297) Oops: (null) due to oops @ 0x791752c5

Quoted 15 lines
CPU: 0 PID: 297 Comm: ls Not tainted 3.13.0+ #1
Hardware name: Dell Computer Corporation Dimension 2350/07W080, BIOS A01 12/17/2002
task: c5298c30 ti: c520e000 task.ti: c520e000
EIP: 0060:[<791752c5>] EFLAGS: 00010286 CPU: 0
EIP is at assfail+0x2b/0x2d
EAX: 00000071 EBX: c60ba600 ECX: 00000296 EDX: c5299098
ESI: c60ba61c EDI: c60ba600 EBP: c520fe40 ESP: c520fe2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
CR0: 80050033 CR2: 08f1612c CR3: 4d1f0000 CR4: 000007d0
Stack:
 00000000 79570bc8 79576e28 7956946d 00000031 c520fe70 791ce45f c520fe70
 7917ceb0 c520fec4 c50d5068 c520fe70 c55d8000 00000000 c50d5068 c607ae30
 c60ba600 c520fed4 791cb72b c607af80 c6c01e80 000000d8 c4294000 c520feec
Call Trace:
 [<791ce45f>] xfs_inode_item_format+0x4a/0x1c5

It's not clear to me that there's anything wrong with the inode log item structure, so I need to know what iovec we tripped over here. Can you post the disassembly of this function so we can see which call to xlog_prepare_iovec tripped the assert? i.e.:

.....

Quoted 1 line

OK, I had to generate a new crash for this, so pardon the dust:

Actually, you don't ned to crash a kernel to do this. Just run 'gdb vmlinux' from your build directory....

Quoted 18 lines

crash> gdb disass /m xfs_inode_item_format
Dump of assembler code for function xfs_inode_item_format:

367     {
   0x791ce415 <+0>:     push   %ebp
   0x791ce416 <+1>:     mov    %esp,%ebp
   0x791ce418 <+3>:     push   %edi
   0x791ce419 <+4>:     push   %esi
   0x791ce41a <+5>:     push   %ebx
   0x791ce41b <+6>:     sub    $0x1c,%esp
   0x791ce41e <+9>:     lea    %ds:0x0(%esi,%eiz,1),%esi
   0x791ce423 <+14>:    mov    %eax,-0x1c(%ebp)
   0x791ce426 <+17>:    mov    %edx,%ebx

368 struct xfs_inode_log_item *iip = INODE_ITEM(lip);

369             struct xfs_inode        *ip = iip->ili_inode;
   0x791ce428 <+19>:    mov    0x44(%eax),%eax
   0x791ce42b <+22>:    mov    %eax,-0x14(%ebp)
Quoted 8 lines

370 struct xfs_inode_log_format *ilf;

371             struct xfs_log_iovec    *vecp = NULL;
   0x791ce42e <+25>:    movl   $0x0,-0x10(%ebp)

372
373 ilf = xlog_prepare_iovec(lv, &vecp, XLOG_REG_TYPE_IFORMAT);

374             ilf->ilf_type = XFS_LI_INODE;
   0x791ce464 <+79>:    movw   $0x123b,(%esi)
Quoted 6 lines

Ok, so xfs_inode_item_format+0x4a is inside the very first call to preapre the ilf structure. That tells us that the initial xfs_log_vec/xfs_log_iovec array are resulting in an unaligned buffer.

Can you try the patch below, Michael?

This patch still works fine. This reply is only in hopes that the patch doesn't get lost for being in a subthread of a [NOISE] post. I'm having to use this patch quite a bit in order to use XFS at all on 32-bit x86.

Note that I'm questioning some of the results I'm getting from git, so I both upgraded git and reworked my kernel git clone. Should this issue go away as a result, I'll post it long before 3.14.0 is released. [I'm not understanding why git merge xfs-oss/master is consistently "Already up-to-date", or why a span of 12 commits in git log takes 13 steps and 3100 revisions to bisect. It could all be OK, but...]

Thanks!

Michael

Quoted 67 lines

Cheers,

Dave.

Signature · 20 lines
Dave Chinner
[email protected]

xfs: ensure correct log item buffer alignment

From: Dave Chinner <[email protected]>

On 32 bit platforms, the log item vector headers are not 64 bit
aligned or sized. hence if we don't take care to align them
correctly or pad the buffer appropriately for 8 byte alignment, we
can end up with alignment issues when accessing the user buffer
directly as a structure.

To solve this, simply pad the buffer headers to 64 bit offset so
that the data section is always 8 byte aligned.

Signed-off-by: Dave Chinner <[email protected]>
---
fs/xfs/xfs_log_cil.c | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
fs/xfs/xfs_log_cil.c +15 -4
index cdebd83..4ef6fdb 100644
@@ -205,16 +205,25 @@ xlog_cil_insert_format_items(
 		/*
 		 * We 64-bit align the length of each iovec so that the start
 		 * of the next one is naturally aligned.  We'll need to
-		 * account for that slack space here.
+		 * account for that slack space here. Then round nbytes up
+		 * to 64-bit alignment so that the initial buffer alignment is
+		 * easy to calculate and verify.
 		 */
 		nbytes += niovecs * sizeof(uint64_t);
+		nbytes = round_up(nbytes, sizeof(uint64_t));
 
 		/* grab the old item if it exists for reservation accounting */
 		old_lv = lip->li_lv;
 
-		/* calc buffer size */
-		buf_size = sizeof(struct xfs_log_vec) + nbytes +
-				niovecs * sizeof(struct xfs_log_iovec);
+		/*
+		 * The data buffer needs to start 64-bit aligned, so round up
+		 * that space to ensure we can align it appropriately and not
+		 * overrun the buffer.
+		 */
+		buf_size = nbytes +
+			   round_up((sizeof(struct xfs_log_vec) +
+				     niovecs * sizeof(struct xfs_log_iovec)),
+				    sizeof(uint64_t));
 
 		/* compare to existing item size */
 		if (lip->li_lv && buf_size <= lip->li_lv->lv_size) {
@@ -251,6 +260,8 @@ xlog_cil_insert_format_items(
 		/* The allocated data region lies beyond the iovec region */
 		lv->lv_buf_len = 0;
 		lv->lv_buf = (char *)lv + buf_size - nbytes;
+		ASSERT(IS_ALIGNED((unsigned long)lv->lv_buf, sizeof(uint64_t)));
+
 		lip->li_ops->iop_format(lip, lv);

insert:

Quoted 2 lines
		ASSERT(lv->lv_buf_len <= nbytes);

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
Quoted 6 lines · Michael L. Semon
Quoted 5 lines · Dave Chinner

Ok, so xfs_inode_item_format+0x4a is inside the very first call to preapre the ilf structure. That tells us that the initial xfs_log_vec/xfs_log_iovec array are resulting in an unaligned buffer.

Quoted 6 lines
Quoted 1 line

Can you try the patch below, Michael?

This patch still works fine. This reply is only in hopes that the patch doesn't get lost for being in a subthread of a [NOISE] post. I'm having to use this patch quite a bit in order to use XFS at all on 32-bit x86.

It's in xfs-fixes-for-3.14-rc3 anf for-next right now. It'll go upstream in the next couple of days.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs