Skip to contentSkip to conversation

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length), file: fs/xfs/xfs_trans_buf.c, line: 568

DISCUSSION Applied Brian Foster 6 participants 19 messages Aug 22, 2013 – Sep 16, 2013 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),
file: fs/xfs/xfs_trans_buf.c, line: 568

------------[ cut here ]------------
kernel BUG at fs/xfs/xfs_message.c:108!
invalid opcode: 0000 [#1] SMP
Modules linked in: xfs libcrc32c fuse ebtable_nat

nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle bnep nf_conntrack_ipv4 nf_defrag_ipv4 bluetooth xt_conntrack nf_conntrack rfkill ebtable_filter ebtables ip6table_filter ip6_tables snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer snd joydev soundcore i2c_piix4 pcspkr mperf virtio_balloon floppy uinput qxl drm_kms_helper ttm drm virtio_blk virtio_net i2c_core

CPU: 0 PID: 1419 Comm: fsstress Not tainted 3.11.0-rc1+ #10
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff8800d65b5dc0 ti: ffff8800d10ba000 task.ti: ffff8800d10ba000
RIP: 0010:[<ffffffffa02b8812>]  [<ffffffffa02b8812>] assfail+0x22/0x30 [xfs]
RSP: 0018:ffff8800d10bb998  EFLAGS: 00010292
RAX: 000000000000006b RBX: ffff8800d67be3a0 RCX: 0000000000000000
RDX: ffff88011fc0ee48 RSI: ffff88011fc0d038 RDI: ffff88011fc0d038
RBP: ffff8800d10bb998 R08: 0000000000000000 R09: 000000000000020a
R10: ffffffff81858260 R11: 0000000000000209 R12: ffff8800d165d500
R13: ffff8800d1158980 R14: 0000000000001007 R15: ffff8800d1cb8300
FS:  00007f1efd2ce740(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1ef80fb018 CR3: 0000000036edb000 CR4: 00000000000006f0
Stack:
 ffff8800d10bb9e8 ffffffffa031d549 000000fc24a6f000 00000e20000000d3
 ffff8800d10bb9f8 ffff8800d67c3040 ffff8800d1cb8208 ffff8800d1cb81e8
 ffff8800d67c3000 ffff8800d1cb8300 ffff8800d10bba48 ffffffffa02e7c1c
Call Trace:
 [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
 [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
 [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
 [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]
 [<ffffffffa02ec965>] xfs_dir_createname+0x1d5/0x1e0 [xfs]
 [<ffffffffa02c1607>] ? kmem_alloc+0x67/0xf0 [xfs]
 [<ffffffffa02becb9>] xfs_symlink+0x619/0xa20 [xfs]
 [<ffffffff811abad6>] ? _d_rehash+0x36/0x40
 [<ffffffff8119f498>] ? __lookup_hash+0x38/0x50
 [<ffffffff8119f4c9>] ? lookup_hash+0x19/0x20
 [<ffffffff811a21ee>] ? kern_path_create+0x8e/0x170
 [<ffffffffa02b5e5c>] xfs_vn_symlink+0x5c/0xe0 [xfs]
 [<ffffffff811a3939>] vfs_symlink+0x99/0x100
 [<ffffffff811a59d6>] SyS_symlinkat+0x66/0xd0
 [<ffffffff811a5a56>] SyS_symlink+0x16/0x20
 [<ffffffff81645442>] system_call_fastpath+0x16/0x1b
Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f1 41 89 d0 48

c7 c6 70 50 33 a0 48 89 fa 31 c0 48 89 e5 31 ff e8 de fb ff ff <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 RIP [<ffffffffa02b8812>] assfail+0x22/0x30 [xfs]
RSP <ffff8800d10bb998>

---[ end trace 9578edaae955ff56 ]---

I repeated the test on a crc=0 fs (with -isize=512) and could not reproduce during fsstress. I let it populate to about 10GB and ultimately hit the same assert on unlink during a post-test cleanup:

crc=0
rm -rf /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),
file: fs/xfs/xfs_trans_buf.c, line: 568

------------[ cut here ]------------
kernel BUG at fs/xfs/xfs_message.c:108!
invalid opcode: 0000 [#1] SMP
Modules linked in: xfs libcrc32c fuse ebtable_nat

nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables bnep bluetooth rfkill ip6table_filter ip6_tables snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer snd soundcore joydev pcspkr virtio_balloon i2c_piix4 floppy mperf uinput qxl drm_kms_helper ttm drm virtio_net virtio_blk i2c_core

CPU: 1 PID: 2198 Comm: rm Not tainted 3.11.0-rc1+ #10
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff8801161ec650 ti: ffff8800c803e000 task.ti: ffff8800c803e000
RIP: 0010:[<ffffffffa02c6812>]  [<ffffffffa02c6812>] assfail+0x22/0x30 [xfs]
RSP: 0018:ffff8800c803fa98  EFLAGS: 00010292
RAX: 000000000000006b RBX: ffff8801029a6e80 RCX: 0000000000000000
RDX: ffff88011fc8ee48 RSI: ffff88011fc8d038 RDI: ffff88011fc8d038
RBP: ffff8800c803fa98 R08: 0000000000000000 R09: 0000000000000209
R10: ffffffff81858260 R11: 0000000000000208 R12: ffff8800302bd200
R13: ffff8800d25e0850 R14: 000000000000122f R15: ffff8800d271f010
FS:  00007f28ef9bf740(0000) GS:ffff88011fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000153a000 CR3: 00000000b1fd3000 CR4: 00000000000006e0
Stack:
 ffff8800c803fae8 ffffffffa032b549 00800201008006cc 000000100185febe
 ffffffffa033fcb0 ffff8800ade0c010 ffff8800ade0c000 ffff8800d3c2b9e0
 ffff8800d25e0850 ffff8800d271f010 ffff8800c803fb58 ffffffffa02f61ff
Call Trace:
 [<ffffffffa032b549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02f61ff>] xfs_da3_node_unbalance+0xef/0x1d0 [xfs]
 [<ffffffffa02f98b0>] xfs_da3_join+0x240/0x290 [xfs]
 [<ffffffffa030659b>] xfs_dir2_node_removename+0x69b/0x8b0 [xfs]
 [<ffffffffa02e16ce>] ? xfs_bmap_last_extent+0x6e/0xb0 [xfs]
 [<ffffffffa02fa5b5>] xfs_dir_removename+0x195/0x1a0 [xfs]
 [<ffffffffa0310b69>] xfs_remove+0x2a9/0x410 [xfs]
 [<ffffffffa02c3ca2>] xfs_vn_unlink+0x52/0xa0 [xfs]
 [<ffffffff811a260e>] vfs_unlink+0x9e/0x110
 [<ffffffff811a2821>] do_unlinkat+0x1a1/0x230
 [<ffffffff811a592b>] SyS_unlinkat+0x1b/0x40
 [<ffffffff81645442>] system_call_fastpath+0x16/0x1b
Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f1 41 89 d0 48

c7 c6 70 30 34 a0 48 89 fa 31 c0 48 89 e5 31 ff e8 de fb ff ff <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 RIP [<ffffffffa02c6812>] assfail+0x22/0x30 [xfs]
RSP <ffff8800c803fa98>

---[ end trace 3ef54f36db3ba0c5 ]---

Info on the crc=0 fs is as follows:

meta-data=/dev/vdb               isize=512    agcount=4, agsize=6553600 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=26214400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=12800, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Brian

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 9 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

Eeek.

So both crashes are directory related. What is the top XFS kernel commit for these tests?

Have you seen this on earlier versions of the kernel?

Thanks,

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 8 lines · Mark Tinguely
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 8 lines
Quoted 2 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

Eeek.

So both crashes are directory related. What is the top XFS kernel commit for these tests?

Hi Mark,

It was the latest from at some point yesterday. I don't think I've pulled since, so I'm at:

3e3c51ce xfs: add xfs sb v4 support for dirent filetype field

Quoted 2 lines

Have you seen this on earlier versions of the kernel?

Well I hit an issue first on my dev. branch for finobt hacking, which is currently based on a slightly older commit:

2c2bcc07 xfs: call roundup_64() to calculate the min_logblks

Unfortunately I don't have the precise error/stack from that failure on hand to double check whether it's the exact same failure. I can try to regenerate it a bit later today.

Brian

Quoted 3 lines

Thanks,

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 31 lines · Brian Foster
Quoted 8 lines · Mark Tinguely
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 8 lines
Quoted 2 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

Eeek.

So both crashes are directory related. What is the top XFS kernel commit for these tests?

Hi Mark,

It was the latest from at some point yesterday. I don't think I've pulled since, so I'm at:

3e3c51ce xfs: add xfs sb v4 support for dirent filetype field

Quoted 2 lines

Have you seen this on earlier versions of the kernel?

Well I hit an issue first on my dev. branch for finobt hacking, which is currently based on a slightly older commit:

Quoted 12 lines

2c2bcc07 xfs: call roundup_64() to calculate the min_logblks

Unfortunately I don't have the precise error/stack from that failure on hand to double check whether it's the exact same failure. I can try to regenerate it a bit later today.

Brian

Quoted 3 lines

Thanks,

--Mark.

Good, I just want to make sure the common directory mods for the file type patch was not the cause. I will try to recreate it too.

Thanks.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 39 lines · Mark Tinguely
Quoted 38 lines · Brian Foster
Quoted 9 lines · Mark Tinguely
Quoted 8 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on
a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 8 lines
Quoted 2 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

Eeek.

So both crashes are directory related. What is the top XFS kernel commit for these tests?

Hi Mark,

It was the latest from at some point yesterday. I don't think I've pulled since, so I'm at:

3e3c51ce xfs: add xfs sb v4 support for dirent filetype field

Quoted 2 lines

Have you seen this on earlier versions of the kernel?

Well I hit an issue first on my dev. branch for finobt hacking, which is currently based on a slightly older commit:

2c2bcc07 xfs: call roundup_64() to calculate the min_logblks

Unfortunately I don't have the precise error/stack from that failure on hand to double check whether it's the exact same failure. I can try to regenerate it a bit later today.

Quoted 9 lines
Quoted 6 lines

Brian

Quoted 3 lines

Thanks,

--Mark.

Good, I just want to make sure the common directory mods for the file type patch was not the cause. I will try to recreate it too.

And just to confirm, I can hit the assert from both tests on a 2c2bcc07 kernel (pre-dirent ftype change).

Brian

Quoted 9 lines

Thanks.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 46 lines · Brian Foster
Quoted 45 lines · Mark Tinguely
Quoted 44 lines · Brian Foster
Quoted 9 lines · Mark Tinguely
Quoted 8 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on
a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 8 lines
Quoted 2 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

Eeek.

So both crashes are directory related. What is the top XFS kernel commit for these tests?

Hi Mark,

It was the latest from at some point yesterday. I don't think I've pulled since, so I'm at:

3e3c51ce xfs: add xfs sb v4 support for dirent filetype field

Quoted 2 lines

Have you seen this on earlier versions of the kernel?

Well I hit an issue first on my dev. branch for finobt hacking, which is currently based on a slightly older commit:

2c2bcc07 xfs: call roundup_64() to calculate the min_logblks

Unfortunately I don't have the precise error/stack from that failure on hand to double check whether it's the exact same failure. I can try to regenerate it a bit later today.

Brian

Quoted 3 lines

Thanks,

--Mark.

Quoted 8 lines
Quoted 3 lines

Good, I just want to make sure the common directory mods for the file type patch was not the cause. I will try to recreate it too.

And just to confirm, I can hit the assert from both tests on a 2c2bcc07 kernel (pre-dirent ftype change).

Brian

Suspected as much, but thank-you for the confirmation.

I guess the next bi-section would be Linux 3.11 to make sure the user/kernel sync did not introduce anything. I thought those moves were clean.

I will shut up and try to help bisect.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
In reply to Brian Foster
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 7 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 5 lines
 [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
 [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
 [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
 [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]

During a split.

Easily reproduced with "seq 200000 | xargs touch" as Michael Semon reported last week.

The fix demonstrates my concerns about modifying directory code - the CRC changes missed a *fundamental* directory format definition, and we've only just tripped over it....

Quoted 3 lines

rm -rf /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 4 lines
 [<ffffffffa032b549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02f61ff>] xfs_da3_node_unbalance+0xef/0x1d0 [xfs]
 [<ffffffffa02f98b0>] xfs_da3_join+0x240/0x290 [xfs]
 [<ffffffffa030659b>] xfs_dir2_node_removename+0x69b/0x8b0 [xfs]

During a merge. Not sure why that is happening on a v4 filesystem. V5 filesystem, yes, due to the above bug but v4 should not be affected.

Cheers,

Dave.

Signature · 16 lines
Dave Chinner
[email protected]

xfs: fix calculation of the number of node entries in a dir3 node

From: Dave Chinner <[email protected]>

The calculation doesn't take into account the size of the dir v3
header, so overestimates the hash entries in a node. This causes
directory buffer overruns when splitting and merging nodes.

Signed-off-by: Dave Chinner <[email protected]>
---
 fs/xfs/xfs_da_btree.h | 11 +++++++++--
 fs/xfs/xfs_dir2.c     | 16 ++++++++++------
 2 files changed, 19 insertions(+), 8 deletions(-)
fs/xfs/xfs_da_btree.h +9 -2
index 8cdc77b..b1f2679 100644
@@ -133,12 +133,19 @@ extern void xfs_da3_node_hdr_to_disk(struct xfs_da_intnode *to,
 				     struct xfs_da3_icnode_hdr *from);
 
 static inline int
-xfs_da3_node_hdr_size(struct xfs_da_intnode *dap)
+__xfs_da3_node_hdr_size(bool v3)
 {
-	if (dap->hdr.info.magic == cpu_to_be16(XFS_DA3_NODE_MAGIC))
+	if (v3)
 		return sizeof(struct xfs_da3_node_hdr);
 	return sizeof(struct xfs_da_node_hdr);
 }
+static inline int
+xfs_da3_node_hdr_size(struct xfs_da_intnode *dap)
+{
+	bool	v3 = dap->hdr.info.magic == cpu_to_be16(XFS_DA3_NODE_MAGIC);
+
+	return __xfs_da3_node_hdr_size(v3);
+}
 
 static inline struct xfs_da_node_entry *
 xfs_da3_node_tree_p(struct xfs_da_intnode *dap)
fs/xfs/xfs_dir2.c +10 -6
index d3ff96c..edf203a 100644
@@ -90,6 +90,9 @@ void
 xfs_dir_mount(
 	xfs_mount_t	*mp)
 {
+	int	nodehdr_size;
+
+
 	ASSERT(xfs_sb_version_hasdirv2(&mp->m_sb));
 	ASSERT((1 << (mp->m_sb.sb_blocklog + mp->m_sb.sb_dirblklog)) <=
 	       XFS_MAX_BLOCKSIZE);
@@ -98,12 +101,13 @@ xfs_dir_mount(
 	mp->m_dirdatablk = xfs_dir2_db_to_da(mp, XFS_DIR2_DATA_FIRSTDB(mp));
 	mp->m_dirleafblk = xfs_dir2_db_to_da(mp, XFS_DIR2_LEAF_FIRSTDB(mp));
 	mp->m_dirfreeblk = xfs_dir2_db_to_da(mp, XFS_DIR2_FREE_FIRSTDB(mp));
-	mp->m_attr_node_ents =
-		(mp->m_sb.sb_blocksize - (uint)sizeof(xfs_da_node_hdr_t)) /
-		(uint)sizeof(xfs_da_node_entry_t);
-	mp->m_dir_node_ents =
-		(mp->m_dirblksize - (uint)sizeof(xfs_da_node_hdr_t)) /
-		(uint)sizeof(xfs_da_node_entry_t);
+
+	nodehdr_size = __xfs_da3_node_hdr_size(xfs_sb_version_hascrc(&mp->m_sb));
+	mp->m_attr_node_ents = (mp->m_sb.sb_blocksize - nodehdr_size) /
+				(uint)sizeof(xfs_da_node_entry_t);
+	mp->m_dir_node_ents = (mp->m_dirblksize - nodehdr_size) /
+				(uint)sizeof(xfs_da_node_entry_t);
+
 	mp->m_dir_magicpct = (mp->m_dirblksize * 37) / 100;
 	if (xfs_sb_version_hasasciici(&mp->m_sb))
 		mp->m_dirnameops = &xfs_ascii_ci_nameops;

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author Tested-by
#7 lore raw
Quoted 39 lines · Dave Chinner
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 7 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 5 lines
 [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
 [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
 [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
 [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]

During a split.

Easily reproduced with "seq 200000 | xargs touch" as Michael Semon reported last week.

The fix demonstrates my concerns about modifying directory code - the CRC changes missed a *fundamental* directory format definition, and we've only just tripped over it....

Quoted 3 lines

rm -rf /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 9 lines
Quoted 4 lines
 [<ffffffffa032b549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02f61ff>] xfs_da3_node_unbalance+0xef/0x1d0 [xfs]
 [<ffffffffa02f98b0>] xfs_da3_join+0x240/0x290 [xfs]
 [<ffffffffa030659b>] xfs_dir2_node_removename+0x69b/0x8b0 [xfs]

During a merge. Not sure why that is happening on a v4 filesystem. V5 filesystem, yes, due to the above bug but v4 should not be affected.

Interesting, thanks Dave. FWIW, I no longer reproduce the assert in either scenario with this patch applied. I also don't see how it would make a difference for a v4 superblock filesystem. Perhaps that particular test was bogus. I haven't heard if Mark happened to reproduce that one. Regardless, consider it:

(xfs: fix calculation of the number of node entries in a dir3 node)

Brian

Quoted 4 lines

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
Quoted 25 lines · Brian Foster
Quoted 24 lines · Dave Chinner
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 7 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 5 lines
  [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
  [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
  [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
  [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
  [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]
Quoted 8 lines
Quoted 8 lines

During a split.

Easily reproduced with "seq 200000 | xargs touch" as Michael Semon reported last week.

The fix demonstrates my concerns about modifying directory code - the CRC changes missed a *fundamental* directory format definition, and we've only just tripped over it....

I agree. As we see here, bugs in common directory code effect all filesystems. It may not matter if the feature the code was written for is enabled or not.

Quoted 22 lines
Quoted 6 lines
Quoted 3 lines

rm -rf /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 9 lines
Quoted 4 lines
  [<ffffffffa032b549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
  [<ffffffffa02f61ff>] xfs_da3_node_unbalance+0xef/0x1d0 [xfs]
  [<ffffffffa02f98b0>] xfs_da3_join+0x240/0x290 [xfs]
  [<ffffffffa030659b>] xfs_dir2_node_removename+0x69b/0x8b0 [xfs]

During a merge. Not sure why that is happening on a v4 filesystem. V5 filesystem, yes, due to the above bug but v4 should not be affected.

Interesting, thanks Dave. FWIW, I no longer reproduce the assert in either scenario with this patch applied. I also don't see how it would make a difference for a v4 superblock filesystem. Perhaps that particular test was bogus. I haven't heard if Mark happened to reproduce that one. Regardless, consider it:

Quoted 3 lines

(xfs: fix calculation of the number of node entries in a dir3 node)

I got the XFS v4 to assert on the remove in Linux 3.10 and 3.11.

With the patch, a shorter test on Linux 3.10 did not assert. I will do the full test on Linux 3.10/3.11, review and report back.

Quoted 6 lines

Brian

Quoted 3 lines

Cheers,

Dave.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
Quoted 31 lines · Mark Tinguely
Quoted 30 lines · Brian Foster
Quoted 29 lines · Dave Chinner
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 7 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 5 lines
 [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
 [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
 [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
 [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]

During a split.

Easily reproduced with "seq 200000 | xargs touch" as Michael Semon reported last week.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

The fix demonstrates my concerns about modifying directory code - the CRC changes missed a *fundamental* directory format definition, and we've only just tripped over it....

I agree. As we see here, bugs in common directory code effect all filesystems. It may not matter if the feature the code was written for is enabled or not.

Well, this is *only* a v5 bug. The fact is, the only difference the change I made makes to v4 filesystems is that it removed the typedef from the sizeof calculation. On my test systems, the value mp->m_dir_node_ents is identical for v4 filesystems with or without the patch applied.....

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

During a merge. Not sure why that is happening on a v4 filesystem. V5 filesystem, yes, due to the above bug but v4 should not be affected.

Interesting, thanks Dave. FWIW, I no longer reproduce the assert in either scenario with this patch applied. I also don't see how it would make a difference for a v4 superblock filesystem. Perhaps that particular test was bogus. I haven't heard if Mark happened to reproduce that one. Regardless, consider it:

Quoted 5 lines
Quoted 3 lines

(xfs: fix calculation of the number of node entries in a dir3 node)

I got the XFS v4 to assert on the remove in Linux 3.10 and 3.11.

Did you test 3.9 - before the crc changes were made to the filesystem? i.e. if an invalid mp->m_dir_node_ents value is the real cause of the v4 filesystem problem, then it should reproduce on just about every kernel we chose to test.

Quoted 2 lines

With the patch, a shorter test on Linux 3.10 did not assert. I will do the full test on Linux 3.10/3.11, review and report back.

Because nobody can explain why this patch would fix a problem on a v4 filesystem, we need more triage of the v4 problem needs to be done. I haven't been able to reproduce the unlink issue (and don't have time to do everything), so could you triage the problem further, Mark? We really need to understand the root cause of the problem on v4 filesystems so we can determine what the impact of it is...

Cheers,

Dave.

Quoted 10 lines
Quoted 7 lines

Brian

Quoted 3 lines

Cheers,

Dave.

--Mark.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#10 lore raw
Quoted 78 lines · Dave Chinner
Quoted 31 lines · Mark Tinguely
Quoted 30 lines · Brian Foster
Quoted 29 lines · Dave Chinner
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 7 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 5 lines
  [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
  [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
  [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
  [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
  [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]

During a split.

Easily reproduced with "seq 200000 | xargs touch" as Michael Semon reported last week.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

The fix demonstrates my concerns about modifying directory code - the CRC changes missed a *fundamental* directory format definition, and we've only just tripped over it....

I agree. As we see here, bugs in common directory code effect all filesystems. It may not matter if the feature the code was written for is enabled or not.

Well, this is *only* a v5 bug. The fact is, the only difference the change I made makes to v4 filesystems is that it removed the typedef from the sizeof calculation. On my test systems, the value mp->m_dir_node_ents is identical for v4 filesystems with or without the patch applied.....

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

During a merge. Not sure why that is happening on a v4 filesystem. V5 filesystem, yes, due to the above bug but v4 should not be affected.

Interesting, thanks Dave. FWIW, I no longer reproduce the assert in either scenario with this patch applied. I also don't see how it would make a difference for a v4 superblock filesystem. Perhaps that particular test was bogus. I haven't heard if Mark happened to reproduce that one. Regardless, consider it:

Quoted 5 lines
Quoted 3 lines

(xfs: fix calculation of the number of node entries in a dir3 node)

I got the XFS v4 to assert on the remove in Linux 3.10 and 3.11.

Did you test 3.9 - before the crc changes were made to the filesystem? i.e. if an invalid mp->m_dir_node_ents value is the real cause of the v4 filesystem problem, then it should reproduce on just about every kernel we chose to test.

Quoted 2 lines

With the patch, a shorter test on Linux 3.10 did not assert. I will do the full test on Linux 3.10/3.11, review and report back.

Because nobody can explain why this patch would fix a problem on a v4 filesystem, we need more triage of the v4 problem needs to be done. I haven't been able to reproduce the unlink issue (and don't have time to do everything), so could you triage the problem further, Mark? We really need to understand the root cause of the problem on v4 filesystems so we can determine what the impact of it is...

Quoted 3 lines

Cheers,

Dave.

A full test still asserts on the remove with the patched Linux 3.10 - I am about 50% into the retest of Linux 3.10 and then I was planning to move back to Linux 3.9.

kdump did not work, so I have no vmcore and therefore no productive information.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw
Quoted 23 lines · Mark Tinguely
Quoted 22 lines · Dave Chinner
Quoted 21 lines · Mark Tinguely
Quoted 20 lines · Brian Foster
Quoted 19 lines · Dave Chinner
Quoted 17 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and
eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I
hit it through a couple different paths, first while running fsstress on
a CRC enabled filesystem (with otherwise default mkfs options):

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),

Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Directory buffer overrun.

A full test still asserts on the remove with the patched Linux 3.10 - I am about 50% into the retest of Linux 3.10 and then I was planning to move back to Linux 3.9.

kdump did not work, so I have no vmcore and therefore no productive information.

Confirmed the Linux 3.10 asserts. Linux 3.9 does not assert.

I will fix the kdump and try to catch a Linux 3.10 assert.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#12 lore raw
In reply to Dave Chinner
Quoted 24 lines · Dave Chinner
Quoted 7 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

Quoted 7 lines

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 5 lines
 [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
 [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
 [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
 [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]
Quoted 8 lines

During a split.

Easily reproduced with "seq 200000 | xargs touch" as Michael Semon reported last week.

The fix demonstrates my concerns about modifying directory code - the CRC changes missed a *fundamental* directory format definition, and we've only just tripped over it....

Don't fret too much over it. This test was part of coreutils, which is something that I rebuild after a glibc upgrade. Had glibc-2.18 been released six weeks ago, then I would have stumbled onto this XFS issue six weeks ago.

Quoted 11 lines
Quoted 3 lines

rm -rf /mnt/test

XFS: Assertion failed: first <= last && last < BBTOB(bp->b_length),

Directory buffer overrun.

Quoted 4 lines
 [<ffffffffa032b549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
 [<ffffffffa02f61ff>] xfs_da3_node_unbalance+0xef/0x1d0 [xfs]
 [<ffffffffa02f98b0>] xfs_da3_join+0x240/0x290 [xfs]
 [<ffffffffa030659b>] xfs_dir2_node_removename+0x69b/0x8b0 [xfs]
Quoted 7 lines

During a merge. Not sure why that is happening on a v4 filesystem. V5 filesystem, yes, due to the above bug but v4 should not be affected.

Cheers,

Dave.

Your patch looks good, and I even applied it to vanilla 3.10.9, along with Jeff Liu's MAX_LFS_FILESIZE patch. [Murphy's Law states that if I didn't use Jeff's patch, then I would run xfstests generic/308 on accident, leading to a hung umount. Happens every single time.] Both patches applied cleanly to kernels on a 2.8 GHz i686 Pentium 4 PC that was running Slackware 14.0 Linux.

Naturally, seq 200000 | xargs touch was run for v5 and v4 XFS file systems. All was okay. The removal of the populated directory went fine as well.

The v5 file systems were tested using a 3.11-rc7+ git kernel. xfstests was run from the start of generic/ through generic/127; and that went fine. Some of the xfs/* series was run but merely scanned because the v5-output-cleanup patches were not readily available.

The v4 file systems were tested with a patched vanilla 3.10.9 kernel, and some of generic was run, with patched and unpatched kernels showing the same good results, very little difference in timing overall.

Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#13 lore raw
In reply to Dave Chinner
Quoted 10 lines · Dave Chinner

xfs: fix calculation of the number of node entries in a dir3 node

From: Dave Chinner<[email protected]>

The calculation doesn't take into account the size of the dir v3 header, so overestimates the hash entries in a node. This causes directory buffer overruns when splitting and merging nodes.

---

Makes sense to me for the inode v3 assert.

The remove assert with the same test must be in common code; it can be triggered in both versions of the inode (Linux 3.10 - TOT). The assert on the remove may require the test to be run until the filesystem is full before doing the remove.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Applied
#14 lore raw
Quoted 12 lines · Mark Tinguely
Quoted 10 lines · Dave Chinner

xfs: fix calculation of the number of node entries in a dir3 node

From: Dave Chinner<[email protected]>

The calculation doesn't take into account the size of the dir v3 header, so overestimates the hash entries in a node. This causes directory buffer overruns when splitting and merging nodes.

---

Quoted 8 lines

Makes sense to me for the inode v3 assert.

The remove assert with the same test must be in common code; it can be triggered in both versions of the inode (Linux 3.10 - TOT). The assert on the remove may require the test to be run until the filesystem is full before doing the remove.

Applied.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#15 lore raw
In reply to Brian Foster
Quoted 135 lines · Brian Foster

Hi all,

I hit an assert on a debug kernel while beating on some finobt work and eventually reproduced it on unmodified/TOT xfs/xfsprogs as of today. I hit it through a couple different paths, first while running fsstress on a CRC enabled filesystem (with otherwise default mkfs options):

(These tests are running on a 4p, 4GB VM against a 100GB virtio disk, hosted on a single spindle desktop box).

crc=1
fsstress -z -fsymlink=1 -n99999999 -p4 -d /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),
file: fs/xfs/xfs_trans_buf.c, line: 568

------------[ cut here ]------------
kernel BUG at fs/xfs/xfs_message.c:108!
invalid opcode: 0000 [#1] SMP
Modules linked in: xfs libcrc32c fuse ebtable_nat

nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle bnep nf_conntrack_ipv4 nf_defrag_ipv4 bluetooth xt_conntrack nf_conntrack rfkill ebtable_filter ebtables ip6table_filter ip6_tables snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer snd joydev soundcore i2c_piix4 pcspkr mperf virtio_balloon floppy uinput qxl drm_kms_helper ttm drm virtio_blk virtio_net i2c_core

CPU: 0 PID: 1419 Comm: fsstress Not tainted 3.11.0-rc1+ #10
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff8800d65b5dc0 ti: ffff8800d10ba000 task.ti: ffff8800d10ba000
RIP: 0010:[<ffffffffa02b8812>]  [<ffffffffa02b8812>] assfail+0x22/0x30 [xfs]
RSP: 0018:ffff8800d10bb998  EFLAGS: 00010292
RAX: 000000000000006b RBX: ffff8800d67be3a0 RCX: 0000000000000000
RDX: ffff88011fc0ee48 RSI: ffff88011fc0d038 RDI: ffff88011fc0d038
RBP: ffff8800d10bb998 R08: 0000000000000000 R09: 000000000000020a
R10: ffffffff81858260 R11: 0000000000000209 R12: ffff8800d165d500
R13: ffff8800d1158980 R14: 0000000000001007 R15: ffff8800d1cb8300
FS:  00007f1efd2ce740(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1ef80fb018 CR3: 0000000036edb000 CR4: 00000000000006f0
Stack:
  ffff8800d10bb9e8 ffffffffa031d549 000000fc24a6f000 00000e20000000d3
  ffff8800d10bb9f8 ffff8800d67c3040 ffff8800d1cb8208 ffff8800d1cb81e8
  ffff8800d67c3000 ffff8800d1cb8300 ffff8800d10bba48 ffffffffa02e7c1c
Call Trace:
  [<ffffffffa031d549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
  [<ffffffffa02e7c1c>] xfs_da3_node_add+0x11c/0x210 [xfs]
  [<ffffffffa02ea703>] xfs_da3_node_split+0xc3/0x230 [xfs]
  [<ffffffffa02eaa18>] xfs_da3_split+0x1a8/0x410 [xfs]
  [<ffffffffa02f743f>] xfs_dir2_node_addname+0x47f/0xde0 [xfs]
  [<ffffffffa02ec965>] xfs_dir_createname+0x1d5/0x1e0 [xfs]
  [<ffffffffa02c1607>] ? kmem_alloc+0x67/0xf0 [xfs]
  [<ffffffffa02becb9>] xfs_symlink+0x619/0xa20 [xfs]
  [<ffffffff811abad6>] ? _d_rehash+0x36/0x40
  [<ffffffff8119f498>] ? __lookup_hash+0x38/0x50
  [<ffffffff8119f4c9>] ? lookup_hash+0x19/0x20
  [<ffffffff811a21ee>] ? kern_path_create+0x8e/0x170
  [<ffffffffa02b5e5c>] xfs_vn_symlink+0x5c/0xe0 [xfs]
  [<ffffffff811a3939>] vfs_symlink+0x99/0x100
  [<ffffffff811a59d6>] SyS_symlinkat+0x66/0xd0
  [<ffffffff811a5a56>] SyS_symlink+0x16/0x20
  [<ffffffff81645442>] system_call_fastpath+0x16/0x1b
Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f1 41 89 d0 48

c7 c6 70 50 33 a0 48 89 fa 31 c0 48 89 e5 31 ff e8 de fb ff ff<0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 RIP [<ffffffffa02b8812>] assfail+0x22/0x30 [xfs]
RSP<ffff8800d10bb998>

---[ end trace 9578edaae955ff56 ]---

I repeated the test on a crc=0 fs (with -isize=512) and could not reproduce during fsstress. I let it populate to about 10GB and ultimately hit the same assert on unlink during a post-test cleanup:

crc=0
rm -rf /mnt/test

XFS: Assertion failed: first<= last&& last< BBTOB(bp->b_length),
file: fs/xfs/xfs_trans_buf.c, line: 568

------------[ cut here ]------------
kernel BUG at fs/xfs/xfs_message.c:108!
invalid opcode: 0000 [#1] SMP
Modules linked in: xfs libcrc32c fuse ebtable_nat

nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables bnep bluetooth rfkill ip6table_filter ip6_tables snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer snd soundcore joydev pcspkr virtio_balloon i2c_piix4 floppy mperf uinput qxl drm_kms_helper ttm drm virtio_net virtio_blk i2c_core

CPU: 1 PID: 2198 Comm: rm Not tainted 3.11.0-rc1+ #10
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff8801161ec650 ti: ffff8800c803e000 task.ti: ffff8800c803e000
RIP: 0010:[<ffffffffa02c6812>]  [<ffffffffa02c6812>] assfail+0x22/0x30 [xfs]
RSP: 0018:ffff8800c803fa98  EFLAGS: 00010292
RAX: 000000000000006b RBX: ffff8801029a6e80 RCX: 0000000000000000
RDX: ffff88011fc8ee48 RSI: ffff88011fc8d038 RDI: ffff88011fc8d038
RBP: ffff8800c803fa98 R08: 0000000000000000 R09: 0000000000000209
R10: ffffffff81858260 R11: 0000000000000208 R12: ffff8800302bd200
R13: ffff8800d25e0850 R14: 000000000000122f R15: ffff8800d271f010
FS:  00007f28ef9bf740(0000) GS:ffff88011fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000153a000 CR3: 00000000b1fd3000 CR4: 00000000000006e0
Stack:
  ffff8800c803fae8 ffffffffa032b549 00800201008006cc 000000100185febe
  ffffffffa033fcb0 ffff8800ade0c010 ffff8800ade0c000 ffff8800d3c2b9e0
  ffff8800d25e0850 ffff8800d271f010 ffff8800c803fb58 ffffffffa02f61ff
Call Trace:
  [<ffffffffa032b549>] xfs_trans_log_buf+0x89/0x1b0 [xfs]
  [<ffffffffa02f61ff>] xfs_da3_node_unbalance+0xef/0x1d0 [xfs]
  [<ffffffffa02f98b0>] xfs_da3_join+0x240/0x290 [xfs]
  [<ffffffffa030659b>] xfs_dir2_node_removename+0x69b/0x8b0 [xfs]
  [<ffffffffa02e16ce>] ? xfs_bmap_last_extent+0x6e/0xb0 [xfs]
  [<ffffffffa02fa5b5>] xfs_dir_removename+0x195/0x1a0 [xfs]
  [<ffffffffa0310b69>] xfs_remove+0x2a9/0x410 [xfs]
  [<ffffffffa02c3ca2>] xfs_vn_unlink+0x52/0xa0 [xfs]
  [<ffffffff811a260e>] vfs_unlink+0x9e/0x110
  [<ffffffff811a2821>] do_unlinkat+0x1a1/0x230
  [<ffffffff811a592b>] SyS_unlinkat+0x1b/0x40
  [<ffffffff81645442>] system_call_fastpath+0x16/0x1b
Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f1 41 89 d0 48

c7 c6 70 30 34 a0 48 89 fa 31 c0 48 89 e5 31 ff e8 de fb ff ff<0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 RIP [<ffffffffa02c6812>] assfail+0x22/0x30 [xfs]
RSP<ffff8800c803fa98>

---[ end trace 3ef54f36db3ba0c5 ]---

Info on the crc=0 fs is as follows:

meta-data=/dev/vdb isize=512 agcount=4, agsize=6553600 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0
data = bsize=4096 blocks=26214400, imaxpct=25
= sunit=0 swidth=0 blks

Quoted 7 lines
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=12800, version=2

= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

Brian

FYI:

The second (rm version) of the test bisects to the patch:

commit f5ea110044fa858925a880b4fa9f551bfa2dfc38

xfs: add CRCs to dir2/da node blocks

---

The secret to tripping over the bug is run the test until fsstress fills the filesystem before removing the files. So an error handling?

I use the test:

#!/bin/sh

ltp/fsstress -z -s 1378390208 -fsymlink=1 -n9999999 -p4 -d /test2 cd /test2
sync
rm -rf *

If your filesystem is smaller, decrease the -n to make the test faster.

I have still not gotten a core, though Michael Semon sent one.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#16 lore raw
Quoted 8 lines · Mark Tinguely

The secret to tripping over the bug is run the test until fsstress fills the filesystem before removing the files. So an error handling?

I use the test:

#!/bin/sh

Quoted 8 lines

ltp/fsstress -z -s 1378390208 -fsymlink=1 -n9999999 -p4 -d /test2 cd /test2
sync
rm -rf *

If your filesystem is smaller, decrease the -n to make the test faster.

I have still not gotten a core, though Michael Semon sent one.

It would be useful if we could wire this up for xfstests

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#17 lore raw
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · Mark Tinguely

The secret to tripping over the bug is run the test until fsstress fills the filesystem before removing the files. So an error handling?

I use the test:

#!/bin/sh

ltp/fsstress -z -s 1378390208 -fsymlink=1 -n9999999 -p4 -d /test2 cd /test2
sync
rm -rf *

Quoted 6 lines
Quoted 3 lines

If your filesystem is smaller, decrease the -n to make the test faster.

I have still not gotten a core, though Michael Semon sent one.

It would be useful if we could wire this up for xfstests

Nod

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#18 lore raw
In reply to Christoph Hellwig
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · Mark Tinguely

The secret to tripping over the bug is run the test until fsstress fills the filesystem before removing the files. So an error handling?

I use the test:

#!/bin/sh

ltp/fsstress -z -s 1378390208 -fsymlink=1 -n9999999 -p4 -d /test2 cd /test2
sync
rm -rf *

Quoted 5 lines
Quoted 3 lines

If your filesystem is smaller, decrease the -n to make the test faster.

I have still not gotten a core, though Michael Semon sent one.

It would be useful if we could wire this up for xfstests

Just set it up accurately because it takes a long time. It takes a
while to create the links. Additionally, fsstress will keep iterating after the FS is full, and all those extra iterations take time.

In testing on x86, Brian's test will succeed in 10 GB but fail in 11 GB. I don't know if that is the case on x86_64 as well. After the failure case is met and the rm operation hits the assert, continued mount-and-rm operations will also cause the assert to fire, without needing to fill the FS with links again. I don't know when that "bonus condition" stops happening. xfs_repair makes matters worse, IIRC.

That's about all I can add. Thanks!

Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs