failed assertion related to realtime section

DISCUSSION Roman Kononov 3 participants 8 messages Jul 13, 2009 – Aug 6, 2009 2 messages not in the archive

Share & tools

Roman Kononov

author

2009-07-13 22:49 UTC #0 lore raw

Hi,

In 2.6.30.x Linux kernel, the XFS_RT configuration option does not say "This feature is unsupported at this time..." any more. Nevertheless, when enabled, the feature crashes the file system in my setup. Is it expected to be functional?

The attached assertion log happens every time with a clean and empty file system.

Thanks,

Roman Kononov

Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/stuff/base/linux-2.6.30.1/fs/xfs/xfs_bmap.c, line: 5602

------------[ cut here ]------------
kernel BUG at /home/stuff/base/linux-2.6.30.1/fs/xfs/support/debug.c:109!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.1/i2c-adapter/i2c-1/1-002e/vrm
CPU 0
Modules linked in: ib_mthca sata_nv
Pid: 1392, comm: ps Not tainted 2.6.30.1 #5
RIP: 0010:[<ffffffff8039f9da>]  [<ffffffff8039f9da>] assfail+0x1a/0x20
RSP: 0018:ffff8800a207bbe8  EFLAGS: 00010292
RAX: 000000000000007a RBX: 0000000000001fff RCX: 0000000000000000
RDX: ffff880028022000 RSI: 0000000000000046 RDI: ffffffff8071a854
RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000000 R11: ffffffff803cfda0 R12: ffff8800aa296f00
R13: ffff8800aa296f38 R14: ffff8800aa296f58 R15: ffff88013ac1f000
FS:  00007f86318316e0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f862f80f000 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ps (pid: 1392, threadinfo ffff8800a207a000, task ffff8800988fc000)
Stack:
 0008000000000000 ffffffff80345e9c ffff8800a207bc48 ffff8800a207bcc8
 ffff8800aa2971b0 000000000000000e 0000000000000001 0000000000000024
 0000000000000000 0000000000000002 0000000000000001 ffff8800a04ed350
Call Trace:
 [<ffffffff80345e9c>] ? xfs_bunmapi+0xc5c/0x11e0
 [<ffffffff8036dd50>] ? xfs_itruncate_finish+0x2b0/0x5d0
 [<ffffffff8038eae1>] ? xfs_free_eofblocks+0x271/0x2c0
 [<ffffffff80396f50>] ? xfs_file_release+0x10/0x20
 [<ffffffff8029ed22>] ? __fput+0xc2/0x210
 [<ffffffff8029b8d6>] ? filp_close+0x56/0x90
 [<ffffffff8023a7f6>] ? put_files_struct+0x76/0xf0
 [<ffffffff8023c560>] ? do_exit+0x670/0x6e0
 [<ffffffff8029e32e>] ? vfs_write+0x11e/0x170
 [<ffffffff8023c615>] ? do_group_exit+0x45/0xb0
 [<ffffffff8023c692>] ? sys_exit_group+0x12/0x20
 [<ffffffff8020bdeb>] ? system_call_fastpath+0x16/0x1b
Code: 08 01 00 00 00 e8 27 19 03 00 48 83 c4 18 c3 66 90 89 d1 48 83 ec 08 48 89 f2 31 c0 48 89 fe 48 c7 c7 e8 d9 5e 80 e8 06 c7 1a 00 <0f> 0b eb fe 66 90 48 83 ec 28 48 89 1c 24 4c 89 6c 24 18 89 fb
RIP  [<ffffffff8039f9da>] assfail+0x1a/0x20
 RSP <ffff8800a207bbe8>
---[ end trace 97560c5848de56ab ]---

Fixing recursive fault but reboot is needed!

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Olaf Weber

2009-07-15 12:22 UTC #1 lore raw

Quoted 5 lines · Roman Kononov

Hi,
In 2.6.30.x Linux kernel, the XFS_RT configuration option does not say "This feature is unsupported at this time..." any more. Nevertheless, when enabled, the feature crashes the file system in my setup. Is it expected to be functional?

I don't think a kernel oops is the expected result.

Quoted 2 lines

The attached assertion log happens every time with a clean and empty file system.

Can you provide:

- the options used to create the filesystem
- the mount options of the filesystem
- (a summary of) the commands run

Signature · 9 lines

Olaf Weber                 SGI               Phone:  +31(0)30-6696752
                           Veldzigt 2b       Fax:    +31(0)30-6696799
Technical Lead             3454 PW de Meern  Vnet:   955-7151
Storage Software           The Netherlands   Email:  [email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Roman Kononov

author

2009-07-16 00:14 UTC #2 lore raw

Quoted 5 lines · Olaf Weber

Can you provide:
- the options used to create the filesystem
- the mount options of the filesystem
- (a summary of) the commands run

The attached bash script (xfs-doit) produces the attached output (out.txt). The kernel messages are attached as well (rt-assert.txt). The kernel was compiled with the attached configuration (config.tar.bz2). The /dev/sda and /dev/sdb are 75GB SATA drives.

+ uname -a
Linux 10.10.10.102 2.6.30.1.xdi #7 SMP Wed Jul 15 16:04:29 CDT 2009 x86_64 x86_64 x86_64 GNU/Linux
+ cd /tmp
+ fuser -mk /dev/sda
+ umount /dev/sda
umount: /dev/sda: not mounted
+ mkfs.xfs -L test -f -i size=2k -l lazy-count=1 -d size=1g -r size=10g,extsize=32m,rtdev=/dev/sdb /dev/sda

meta-data=/dev/sda               isize=2048   agcount=8, agsize=32768 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2560, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =/dev/sdb               extsz=33554432 blocks=2621440, rtextents=320

+ mkdir -p test
+ mount -o noikeep,largeio,noatime,nodiratime,swalloc,nobarrier,rtdev=/dev/sdb /dev/sda test
+ mkdir test/data test/rltm
+ /usr/sbin/xfs_io -c 'chattr +r' -c 'chattr +t' test/rltm
+ for file in '{0..1}'
+ dd if=/dev/zero of=test/rltm/0 bs=1048576 count=100 seek=10
+ for file in '{0..1}'
+ dd if=/dev/zero of=test/rltm/1 bs=1048576 count=100 seek=10
+ wait
tmp/xfs-doit: line 17: 834 Segmentation fault dd if=/dev/zero of=test/rltm/$file bs=$((1024*1024)) count=100 seek=10
tmp/xfs-doit: line 17: 835 Segmentation fault dd if=/dev/zero of=test/rltm/$file bs=$((1024*1024)) count=100 seek=10
+ umount test
umount: /tmp/test: device is busy
umount: /tmp/test: device is busy
+ exit

XFS: correcting sb_features alignment problem
XFS mounting filesystem sda
Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/stuff/base/linux-2.6.30.1/fs/xfs/xfs_bmap.c, line: 5602

------------[ cut here ]------------
kernel BUG at /home/stuff/base/linux-2.6.30.1/fs/xfs/support/debug.c:109!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/host2/target2:0:0/2:0:0:0/block/sdb/dev
CPU 0
Modules linked in: ib_mthca sata_nv
Pid: 834, comm: dd Not tainted 2.6.30.1.xdi #7
RIP: 0010:[<ffffffff8039f9da>]  [<ffffffff8039f9da>] assfail+0x1a/0x20
RSP: 0018:ffff88013b159ca8  EFLAGS: 00010296
RAX: 000000000000007a RBX: 0000000000007fff RCX: 0000000000000000
RDX: ffff880028022000 RSI: 0000000000000046 RDI: ffffffff8071a854
RBP: 0000000000006e00 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000000 R11: ffffffff803cfda0 R12: ffff88013f5772c0
R13: ffff88013f5772f8 R14: ffff88013f577318 R15: ffff88013b015400
FS:  00007fa7409206e0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fe258d1d000 CR3: 000000013b17b000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dd (pid: 834, threadinfo ffff88013b158000, task ffff88013e828b20)
Stack:
 0008000000000000 ffffffff80345e9c ffff88013b159d08 ffff88013b159d88
 ffff88013f577570 000000000000000e 0000000000006e00 0000000000001286
 0000000000000000 0000000000000002 0000000000000001 ffff88013b1b2030
Call Trace:
 [<ffffffff80345e9c>] ? xfs_bunmapi+0xc5c/0x11e0
 [<ffffffff8036dd50>] ? xfs_itruncate_finish+0x2b0/0x5d0
 [<ffffffff8038eae1>] ? xfs_free_eofblocks+0x271/0x2c0
 [<ffffffff80396f50>] ? xfs_file_release+0x10/0x20
 [<ffffffff8029ed22>] ? __fput+0xc2/0x210
 [<ffffffff8029b8d6>] ? filp_close+0x56/0x90
 [<ffffffff8029b9a1>] ? sys_close+0x91/0xd0
 [<ffffffff8020bdeb>] ? system_call_fastpath+0x16/0x1b
Code: 08 01 00 00 00 e8 27 19 03 00 48 83 c4 18 c3 66 90 89 d1 48 83 ec 08 48 89 f2 31 c0 48 89 fe 48 c7 c7 e8 d9 5e 80 e8 06 c7 1a 00 <0f> 0b eb fe 66 90 48 83 ec 28 48 89 1c 24 4c 89 6c 24 18 89 fb
RIP  [<ffffffff8039f9da>] assfail+0x1a/0x20
 RSP <ffff88013b159ca8>
---[ end trace 88cf42faf5c1e431 ]---
Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/stuff/base/linux-2.6.30.1/fs/xfs/xfs_bmap.c, line: 5602
------------[ cut here ]------------
kernel BUG at /home/stuff/base/linux-2.6.30.1/fs/xfs/support/debug.c:109!
invalid opcode: 0000 [#2] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/host2/target2:0:0/2:0:0:0/block/sdb/dev
CPU 0
Modules linked in: ib_mthca sata_nv
Pid: 835, comm: dd Tainted: G      D    2.6.30.1.xdi #7
RIP: 0010:[<ffffffff8039f9da>]  [<ffffffff8039f9da>] assfail+0x1a/0x20
RSP: 0018:ffff88013b131ca8  EFLAGS: 00010296
RAX: 000000000000007a RBX: 0000000000007fff RCX: 0000000000000000
RDX: ffff880028022000 RSI: 0000000000000046 RDI: ffffffff8071a854
RBP: 0000000000006e00 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000000 R11: ffffffff803cfda0 R12: ffff88013f577680
R13: ffff88013f5776b8 R14: ffff88013f5776d8 R15: ffff88013b015400
FS:  00007fe258d206e0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000037f3ed1060 CR3: 000000013b17a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dd (pid: 835, threadinfo ffff88013b130000, task ffff88013e82ac80)
Stack:
 0008000000000000 ffffffff80345e9c ffff88013b131d08 ffff88013b131d88
 ffff88013f577930 000000000000000e 0000000000006e00 0000000000001286
 0000000000000000 0000000000000002 0000000000000001 ffff88013b1b2368
Call Trace:
 [<ffffffff80345e9c>] ? xfs_bunmapi+0xc5c/0x11e0
 [<ffffffff8036dd50>] ? xfs_itruncate_finish+0x2b0/0x5d0
 [<ffffffff8038eae1>] ? xfs_free_eofblocks+0x271/0x2c0
 [<ffffffff80396f50>] ? xfs_file_release+0x10/0x20
 [<ffffffff8029ed22>] ? __fput+0xc2/0x210
 [<ffffffff8029b8d6>] ? filp_close+0x56/0x90
 [<ffffffff8029b9a1>] ? sys_close+0x91/0xd0
 [<ffffffff8020bdeb>] ? system_call_fastpath+0x16/0x1b
Code: 08 01 00 00 00 e8 27 19 03 00 48 83 c4 18 c3 66 90 89 d1 48 83 ec 08 48 89 f2 31 c0 48 89 fe 48 c7 c7 e8 d9 5e 80 e8 06 c7 1a 00 <0f> 0b eb fe 66 90 48 83 ec 28 48 89 1c 24 4c 89 6c 24 18 89 fb
RIP  [<ffffffff8039f9da>] assfail+0x1a/0x20
 RSP <ffff88013b131ca8>
---[ end trace 88cf42faf5c1e432 ]---

#!/bin/bash
data_dev=/dev/sda
rltm_dev=/dev/sdb
set -x
uname -a
cd /tmp
fuser -mk $data_dev
umount $data_dev
mkfs.xfs -L test -f -i size=2k -l lazy-count=1 -d size=1g -r size=10g,extsize=32m,rtdev=$rltm_dev $data_dev || exit mkdir -p test || exit
mount -o noikeep,largeio,noatime,nodiratime,swalloc,nobarrier,rtdev=$rltm_dev $data_dev test || exit mkdir test/data test/rltm || exit
/usr/sbin/xfs_io -c "chattr +r" -c "chattr +t" test/rltm || exit
for file in {0..1}; do

	dd if=/dev/zero of=test/rltm/$file bs=$((1024*1024)) count=100 seek=10 &

done
wait
umount test || exit
xfs_repair -nv -r $rltm_dev $data_dev || exit
echo "done"

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Roman Kononov

author

2009-08-04 21:04 UTC #4 lore raw

In reply to a message not in the archive

Quoted 3 lines · Olaf Weber

A quick test on an available system
running a (much) older kernel runs to completion, so this appears to be a regression.

With which kernel version did you run the test?

Quoted 1 line

Figuring out exactly when/where this regressed will take time.

I went back to 2.6.23 and the same assfail happened with somewhat different call trace:

Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/rk/linux-2.6.23/fs/xfs/xfs_bmap.c, line: 5523

Call Trace:
 [<ffffffff80356a08>] xfs_bunmapi+0x8e8/0x1060
 [<ffffffff80227d2a>] dequeue_entity+0x7a/0xb0
 [<ffffffff80380608>] xfs_itruncate_finish+0x398/0x5c0
 [<ffffffff803a2323>] xfs_free_eofblocks+0x263/0x2b0
 [<ffffffff803a4838>] xfs_release+0x118/0x1e0
 [<ffffffff803ae42a>] xfs_file_release+0x1a/0x30
 [<ffffffff802827cd>] __fput+0xcd/0x1e0
 [<ffffffff8027f5c4>] filp_close+0x54/0x90
 [<ffffffff80280e3d>] sys_close+0x9d/0x110
 [<ffffffff8020bc9e>] system_call+0x7e/0x83

I tried to go back to 2.6.22.19 and older, and the test did not run at all failing to mount:

XFS: bad version
XFS: SB validate failed

Any suggestions?

Roman

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Eric Sandeen

2009-08-05 03:58 UTC #8 lore raw

In reply to Roman Kononov

Quoted 7 lines · Roman Kononov

Hi,
In 2.6.30.x Linux kernel, the XFS_RT configuration option does not say "This feature is unsupported at this time..." any more. Nevertheless, when enabled, the feature crashes the file system in my setup. Is it expected to be functional?

Quoted 6 lines

The attached assertion log happens every time with a clean and empty file system.
Thanks,
Roman Kononov

While realtime should probably work and not oops (!) I'm curious what your use-case for realtime is?

Thanks,
-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Roman Kononov

author

2009-08-05 23:07 UTC #9 lore raw

Quoted 2 lines · Eric Sandeen

While realtime should probably work and not oops (!) I'm curious what your use-case for realtime is?

In my setup I need mostly sequential read/write access to a number of huge files (>1GB) as well as small ones. My 16-disk system, providing 1GB/s of sequential access data rate, chokes when file fragments are smaller than ~32MB. Using 32MB extents of the regular data section helps, but fragmentation still develops. I hope (maybe I am wrong) that the realtime section with its fixed extents eliminates the fragmentation issue.

Roman

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Roman Kononov

author

2009-08-05 23:21 UTC #6 lore raw

In reply to a message not in the archive

Quoted 36 lines · Olaf Weber

Quoted 6 lines · Roman Kononov
Quoted 3 lines · Olaf Weber
A quick test on an available system
running a (much) older kernel runs to completion, so this appears to be a regression.
With which kernel version did you run the test?
Heavily-patched 2.6.16, which is what I had readily available at that point. But most or all of those patches + additional changes should be in current XFS.
Quoted 22 lines
Quoted 1 line
Figuring out exactly when/where this regressed will take time.
I went back to 2.6.23 and the same assfail happened with somewhat different call trace:
Assertion failed: xfs_trans_get_block_res(tp) > 0, file:
/home/rk/linux-2.6.23/fs/xfs/xfs_bmap.c, line: 5523
Call Trace:
 [<ffffffff80356a08>] xfs_bunmapi+0x8e8/0x1060
 [<ffffffff80227d2a>] dequeue_entity+0x7a/0xb0
 [<ffffffff80380608>] xfs_itruncate_finish+0x398/0x5c0
 [<ffffffff803a2323>] xfs_free_eofblocks+0x263/0x2b0
 [<ffffffff803a4838>] xfs_release+0x118/0x1e0
 [<ffffffff803ae42a>] xfs_file_release+0x1a/0x30
 [<ffffffff802827cd>] __fput+0xcd/0x1e0
 [<ffffffff8027f5c4>] filp_close+0x54/0x90
 [<ffffffff80280e3d>] sys_close+0x9d/0x110
 [<ffffffff8020bc9e>] system_call+0x7e/0x83
I tried to go back to 2.6.22.19 and older, and the test did not run at all failing to mount:

Quoted 9 lines

Quoted 4 lines
XFS: bad version
XFS: SB validate failed
Any suggestions?
For that I had the advantage of being able to just build a fresh XFS filesystem for the purpose. If that is what you're doing as well, look at disabling the lazy-counters option at mkfs time.

I removed lazy-counters as you suggested, and under 2.6.22.19 it mounts successfully. But still crushes in the same place:

Assertion failed: ((tp)->t_blk_res) > 0, file: /home/rk/xbase/linux/linux-2.6.22.19/fs/xfs/xfs_bmap.c, line: 5515

Call Trace: 


 [<ffffffff8034ef86>] xfs_bunmapi+0x8e6/0x1050
 [<ffffffff80208042>] __switch_to+0x42/0x310
 [<ffffffff80377dc8>] xfs_itruncate_finish+0x398/0x5c0
 [<ffffffff803989c1>] xfs_inactive_free_eofblocks+0x221/0x260
 [<ffffffff802782c7>] cache_free_debugcheck+0xc7/0x1d0
 [<ffffffff8039aee7>] xfs_release+0xc7/0x110
 [<ffffffff803a497a>] xfs_file_release+0x1a/0x30
 [<ffffffff8027f02d>] __fput+0xcd/0x1e0
 [<ffffffff8027bf54>] filp_close+0x54/0x90
 [<ffffffff8027d68d>] sys_close+0x9d/0x110
 [<ffffffff80209c0e>] system_call+0x7e/0x83

I will try older versions.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Roman Kononov

author

2009-08-06 00:37 UTC #7 lore raw

In reply to a message not in the archive

Quoted 3 lines · Olaf Weber

A quick test on an available system
running a (much) older kernel runs to completion, so this appears to be a regression.

Did you have the XFS_DEBUG option on? Without it, the filesystem with realtime does not die right away, but hangs or crashes later.

Quoted 7 lines

Quoted 2 lines
With which kernel version did you run the test?
Heavily-patched 2.6.16, which is what I had readily available at that point. But most or all of those patches + additional changes should be in current XFS.

Vanilla 2.6.16.19 without the lazy-counters option:

Assertion failed: ((tp)->t_blk_res) > 0, file: /home/rk/linux-2.6.16.19/fs/xfs/xfs_bmap.c, line: 5452

Call Trace:

<ffffffff801e4b3f>{xfs_bunmapi+1335} <ffffffff8020c5c0>{xfs_itruncate_finish+769} <ffffffff8022be3c>{xfs_inactive_free_eofblocks+420} <ffffffff8022bf40>{xfs_release+190} <ffffffff80232e27>{linvfs_release+26} <ffffffff80169c60>{__fput+189} <ffffffff8016733a>{filp_close+93} <ffffffff801673db>{sys_close+150} <ffffffff8010aa2a>{system_call+126}

It looks like it never worked.

Any hope? :-(

Roman

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs