Skip to contentSkip to conversation

reflink log reservations

DISCUSSION Christoph Hellwig 2 participants 5 messages Feb 4, 2016 – Feb 9, 2016 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi Darrick,

I'm running into the following log reservation assert after a few xfstests runs over NFS. I'll try to understand the reservation calculations, but in case this is still fresh in your head here's a headsup:

generic/167 11s ...

[  640.924891] XFS: Assertion failed: tp->t_blk_res_used <= tp->t_blk_res, file: fs/xfs/xfs_trans.c, line: 315
[  640.925750] ------------[ cut here ]------------
[  640.926164] kernel BUG at fs/xfs/xfs_message.c:113!
[  640.926583] invalid opcode: 0000 [#1] SMP 
[  640.926995] Modules linked in:
[  640.927385] CPU: 3 PID: 3719 Comm: nfsd Not tainted 4.5.0-rc2+ #424
[  640.928036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[  640.928098] task: ffff88007b614380 ti: ffff880077cd4000 task.ti: ffff880077cd4000
[  640.928098] RIP: 0010:[<ffffffff814cdd1d>]  [<ffffffff814cdd1d>] assfail+0x1d/0x20
[  640.928098] RSP: 0018:ffff880077cd72e0  EFLAGS: 00010282
[  640.928098] RAX: 00000000ffffffea RBX: ffff88007afc9130 RCX: 0000000000000021
[  640.928098] RDX: ffff880077cd7208 RSI: 000000000000000a RDI: ffffffff81fdc510
[  640.928098] RBP: ffff880077cd72e0 R08: 0000000000000000 R09: 0000000000000000
[  640.928098] R10: 000000000000000a R11: f000000000000000 R12: ffffffffffffffff
[  640.928098] R13: ffff88007b2cd000 R14: 00000000000a0000 R15: 0000000000000001
[  640.928098] FS:  0000000000000000(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000
[  640.928098] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  640.928098] CR2: 00000000013e10f8 CR3: 0000000075b18000 CR4: 00000000000006e0
[  640.928098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  640.928098] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  640.928098] Stack:
[  640.928098]  ffff880077cd7308 ffffffff814d7d00 ffff880077cd7390 0000000000000001
[  640.928098]  ffff880077cd7390 ffff880077cd7328 ffffffff8146059d ffff88007b2cd000
[  640.928098]  0000000000000020 ffff880077cd7378 ffffffff814617cd ffff880077cd7378
[  640.928098] Call Trace:
[  640.928098]  [<ffffffff814d7d00>] xfs_trans_mod_sb+0x250/0x290
[  640.928098]  [<ffffffff8146059d>] xfs_alloc_ag_vextent+0x28d/0x330
[  640.928098]  [<ffffffff814617cd>] xfs_alloc_vextent+0x70d/0x800
[  640.928098]  [<ffffffff814a7a51>] xfs_refcountbt_alloc_block+0xd1/0x1a0
[  640.928098]  [<ffffffff81480aa6>] __xfs_btree_split+0xb6/0xbd0
[  640.928098]  [<ffffffff814eb6e6>] ? xfs_trans_read_buf_map+0x1e6/0x370
[  640.928098]  [<ffffffff814816a4>] xfs_btree_split+0x34/0xc0
[  640.928098]  [<ffffffff8148287f>] xfs_btree_make_block_unfull+0xef/0x170
[  640.928098]  [<ffffffff814832f9>] xfs_btree_insrec+0x9f9/0xc50
[  640.928098]  [<ffffffff8147d6f4>] ? xfs_btree_read_buf_block+0xa4/0xd0
[  640.928098]  [<ffffffff814ebe1f>] ? xfs_trans_log_buf+0x15f/0x1b0
[  640.928098]  [<ffffffff814835b9>] xfs_btree_insert+0x69/0x1d0
[  640.928098]  [<ffffffff814a509e>] xfs_refcountbt_insert+0x4e/0x90
[  640.928098]  [<ffffffff814a58af>] try_split_right_rcextent+0x17f/0x200
[  640.928098]  [<ffffffff814a6afc>] xfs_refcountbt_adjust_refcount+0x4c/0x100
[  640.928098]  [<ffffffff814a6cac>] xfs_refcount_decrease+0x5c/0xa0
[  640.928098]  [<ffffffff814a6d73>] xfs_refcount_put_extent+0x83/0xb0
[  640.928098]  [<ffffffff8147706f>] xfs_bmap_del_extent+0x57f/0x1160
[  640.928098]  [<ffffffff814d965b>] ? kmem_zone_alloc+0x7b/0x120
[  640.928098]  [<ffffffff814d965b>] ? kmem_zone_alloc+0x7b/0x120
[  640.928098]  [<ffffffff8147b8ef>] ? xfs_bmbt_init_cursor+0x5f/0x190
[  640.928098]  [<ffffffff81478bf5>] xfs_bunmapi+0x845/0x10d0
[  640.928098]  [<ffffffff814c8ad8>] xfs_itruncate_extents+0x218/0x3e0
[  640.928098]  [<ffffffff814c8e8d>] xfs_inactive_truncate+0xbd/0x130
[  640.928098]  [<ffffffff814ca6db>] xfs_inactive+0x1ab/0x1e0
[  640.928098]  [<ffffffff814d3a49>] xfs_fs_evict_inode+0x109/0x160
[  640.928098]  [<ffffffff811d79db>] evict+0xbb/0x180
[  640.928098]  [<ffffffff811d8314>] iput+0x174/0x1e0
[  640.928098]  [<ffffffff811d39be>] d_delete+0x11e/0x160
[  640.928098]  [<ffffffff811ca203>] vfs_unlink+0x133/0x160
[  640.928098]  [<ffffffff811ca74a>] ? lookup_one_len+0xca/0x120
[  640.928098]  [<ffffffff81359897>] nfsd_unlink+0x127/0x210
[  640.928098]  [<ffffffff81364d69>] nfsd4_remove+0x49/0x130
[  640.928098]  [<ffffffff8136593a>] nfsd4_proc_compound+0x31a/0x570
[  640.928098]  [<ffffffff813549d9>] nfsd_dispatch+0x89/0x170
[  640.928098]  [<ffffffff81b9968a>] svc_process_common+0x39a/0x520
[  640.928098]  [<ffffffff81b99960>] svc_process+0x150/0x1a0
[  640.928098]  [<ffffffff8135446a>] nfsd+0xea/0x150
[  640.928098]  [<ffffffff81354380>] ? nfsd_destroy+0x60/0x60
[  640.928098]  [<ffffffff810dedf6>] kthread+0xd6/0xf0
[  640.928098]  [<ffffffff810ded20>] ? kthread_create_on_node+0x170/0x170
[  640.928098]  [<ffffffff81bc8a8f>] ret_from_fork+0x3f/0x70
[  640.928098]  [<ffffffff810ded20>] ? kthread_create_on_node+0x170/0x170

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 6 lines · Christoph Hellwig

Hi Darrick,

I'm running into the following log reservation assert after a few xfstests runs over NFS. I'll try to understand the reservation calculations, but in case this is still fresh in your head here's a headsup:

Hmm, interesting, I haven't seen /that/ one... my guess is that the block reservation needs to be bumped up for the refcount tree.

--D

Quoted 72 lines

generic/167 11s ...

[  640.924891] XFS: Assertion failed: tp->t_blk_res_used <= tp->t_blk_res, file: fs/xfs/xfs_trans.c, line: 315
[  640.925750] ------------[ cut here ]------------
[  640.926164] kernel BUG at fs/xfs/xfs_message.c:113!
[  640.926583] invalid opcode: 0000 [#1] SMP 
[  640.926995] Modules linked in:
[  640.927385] CPU: 3 PID: 3719 Comm: nfsd Not tainted 4.5.0-rc2+ #424
[  640.928036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[  640.928098] task: ffff88007b614380 ti: ffff880077cd4000 task.ti: ffff880077cd4000
[  640.928098] RIP: 0010:[<ffffffff814cdd1d>]  [<ffffffff814cdd1d>] assfail+0x1d/0x20
[  640.928098] RSP: 0018:ffff880077cd72e0  EFLAGS: 00010282
[  640.928098] RAX: 00000000ffffffea RBX: ffff88007afc9130 RCX: 0000000000000021
[  640.928098] RDX: ffff880077cd7208 RSI: 000000000000000a RDI: ffffffff81fdc510
[  640.928098] RBP: ffff880077cd72e0 R08: 0000000000000000 R09: 0000000000000000
[  640.928098] R10: 000000000000000a R11: f000000000000000 R12: ffffffffffffffff
[  640.928098] R13: ffff88007b2cd000 R14: 00000000000a0000 R15: 0000000000000001
[  640.928098] FS:  0000000000000000(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000
[  640.928098] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  640.928098] CR2: 00000000013e10f8 CR3: 0000000075b18000 CR4: 00000000000006e0
[  640.928098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  640.928098] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  640.928098] Stack:
[  640.928098]  ffff880077cd7308 ffffffff814d7d00 ffff880077cd7390 0000000000000001
[  640.928098]  ffff880077cd7390 ffff880077cd7328 ffffffff8146059d ffff88007b2cd000
[  640.928098]  0000000000000020 ffff880077cd7378 ffffffff814617cd ffff880077cd7378
[  640.928098] Call Trace:
[  640.928098]  [<ffffffff814d7d00>] xfs_trans_mod_sb+0x250/0x290
[  640.928098]  [<ffffffff8146059d>] xfs_alloc_ag_vextent+0x28d/0x330
[  640.928098]  [<ffffffff814617cd>] xfs_alloc_vextent+0x70d/0x800
[  640.928098]  [<ffffffff814a7a51>] xfs_refcountbt_alloc_block+0xd1/0x1a0
[  640.928098]  [<ffffffff81480aa6>] __xfs_btree_split+0xb6/0xbd0
[  640.928098]  [<ffffffff814eb6e6>] ? xfs_trans_read_buf_map+0x1e6/0x370
[  640.928098]  [<ffffffff814816a4>] xfs_btree_split+0x34/0xc0
[  640.928098]  [<ffffffff8148287f>] xfs_btree_make_block_unfull+0xef/0x170
[  640.928098]  [<ffffffff814832f9>] xfs_btree_insrec+0x9f9/0xc50
[  640.928098]  [<ffffffff8147d6f4>] ? xfs_btree_read_buf_block+0xa4/0xd0
[  640.928098]  [<ffffffff814ebe1f>] ? xfs_trans_log_buf+0x15f/0x1b0
[  640.928098]  [<ffffffff814835b9>] xfs_btree_insert+0x69/0x1d0
[  640.928098]  [<ffffffff814a509e>] xfs_refcountbt_insert+0x4e/0x90
[  640.928098]  [<ffffffff814a58af>] try_split_right_rcextent+0x17f/0x200
[  640.928098]  [<ffffffff814a6afc>] xfs_refcountbt_adjust_refcount+0x4c/0x100
[  640.928098]  [<ffffffff814a6cac>] xfs_refcount_decrease+0x5c/0xa0
[  640.928098]  [<ffffffff814a6d73>] xfs_refcount_put_extent+0x83/0xb0
[  640.928098]  [<ffffffff8147706f>] xfs_bmap_del_extent+0x57f/0x1160
[  640.928098]  [<ffffffff814d965b>] ? kmem_zone_alloc+0x7b/0x120
[  640.928098]  [<ffffffff814d965b>] ? kmem_zone_alloc+0x7b/0x120
[  640.928098]  [<ffffffff8147b8ef>] ? xfs_bmbt_init_cursor+0x5f/0x190
[  640.928098]  [<ffffffff81478bf5>] xfs_bunmapi+0x845/0x10d0
[  640.928098]  [<ffffffff814c8ad8>] xfs_itruncate_extents+0x218/0x3e0
[  640.928098]  [<ffffffff814c8e8d>] xfs_inactive_truncate+0xbd/0x130
[  640.928098]  [<ffffffff814ca6db>] xfs_inactive+0x1ab/0x1e0
[  640.928098]  [<ffffffff814d3a49>] xfs_fs_evict_inode+0x109/0x160
[  640.928098]  [<ffffffff811d79db>] evict+0xbb/0x180
[  640.928098]  [<ffffffff811d8314>] iput+0x174/0x1e0
[  640.928098]  [<ffffffff811d39be>] d_delete+0x11e/0x160
[  640.928098]  [<ffffffff811ca203>] vfs_unlink+0x133/0x160
[  640.928098]  [<ffffffff811ca74a>] ? lookup_one_len+0xca/0x120
[  640.928098]  [<ffffffff81359897>] nfsd_unlink+0x127/0x210
[  640.928098]  [<ffffffff81364d69>] nfsd4_remove+0x49/0x130
[  640.928098]  [<ffffffff8136593a>] nfsd4_proc_compound+0x31a/0x570
[  640.928098]  [<ffffffff813549d9>] nfsd_dispatch+0x89/0x170
[  640.928098]  [<ffffffff81b9968a>] svc_process_common+0x39a/0x520
[  640.928098]  [<ffffffff81b99960>] svc_process+0x150/0x1a0
[  640.928098]  [<ffffffff8135446a>] nfsd+0xea/0x150
[  640.928098]  [<ffffffff81354380>] ? nfsd_destroy+0x60/0x60
[  640.928098]  [<ffffffff810dedf6>] kthread+0xd6/0xf0
[  640.928098]  [<ffffffff810ded20>] ? kthread_create_on_node+0x170/0x170
[  640.928098]  [<ffffffff81bc8a8f>] ret_from_fork+0x3f/0x70
[  640.928098]  [<ffffffff810ded20>] ? kthread_create_on_node+0x170/0x170
Quoted 4 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 10 lines · Darrick J. Wong
Quoted 6 lines · Christoph Hellwig

Hi Darrick,

I'm running into the following log reservation assert after a few xfstests runs over NFS. I'll try to understand the reservation calculations, but in case this is still fresh in your head here's a headsup:

Hmm, interesting, I haven't seen /that/ one... my guess is that the block reservation needs to be bumped up for the refcount tree.

This seems to fix it for me:

---

Quoted 1 line

From 6e27eb42ece3a25f610690487b403a0091cafa26 Mon Sep 17 00:00:00 2001

From: Christoph Hellwig <[email protected]>
Date: Thu, 4 Feb 2016 13:50:16 +0100
Subject: xfs: account for the refcount btree in the alloc/free log reservation

---
 fs/xfs/libxfs/xfs_trans_resv.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_trans_resv.c +10 -5
index d495f82..f4c2648 100644
@@ -65,12 +65,15 @@ xfs_calc_buf_res(
 
 /*
  * Per-extent log reservation for the allocation btree changes
- * involved in freeing or allocating an extent. When rmap is not enabled,
- * there are only two trees that will be modified (free space trees), and when
- * rmap is enabled there will be three (freespace + rmap trees). The number of
- * blocks reserved is based on the formula:
+ * involved in freeing or allocating an extent.
  *
- * num trees * ((2 blocks/level * max depth) - 1)
+ * There are at least two trees that will be modified (free space trees), when
+ * rmap is enabled there will be an additional rmap tree, and when reflinks
+ * are enabled there will be a refcount btree as well
+ *
+ * The number of  blocks reserved is based on the formula:
+ *
+ *     num trees * ((2 blocks/level * max depth) - 1)
  */
 static uint
 xfs_allocfree_log_count(
@@ -81,6 +84,8 @@ xfs_allocfree_log_count(
 
 	if (xfs_sb_version_hasrmapbt(&mp->m_sb))
 		num_trees++;
+	if (xfs_sb_version_hasreflink(&mp->m_sb))
+		num_trees++;
 
 	return num_ops * num_trees * (2 * mp->m_ag_maxlevels - 1);
 }
Signature · 6 lines
2.1.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 37 lines · Christoph Hellwig
Quoted 10 lines · Darrick J. Wong
Quoted 6 lines · Christoph Hellwig

Hi Darrick,

I'm running into the following log reservation assert after a few xfstests runs over NFS. I'll try to understand the reservation calculations, but in case this is still fresh in your head here's a headsup:

Hmm, interesting, I haven't seen /that/ one... my guess is that the block reservation needs to be bumped up for the refcount tree.

This seems to fix it for me:

---
From 6e27eb42ece3a25f610690487b403a0091cafa26 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Thu, 4 Feb 2016 13:50:16 +0100
Subject: xfs: account for the refcount btree in the alloc/free log reservation

---
 fs/xfs/libxfs/xfs_trans_resv.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
fs/xfs/libxfs/xfs_trans_resv.c +0 -4
index d495f82..f4c2648 100644
@@ -65,12 +65,15 @@ xfs_calc_buf_res(
 
 /*
  * Per-extent log reservation for the allocation btree changes
- * involved in freeing or allocating an extent. When rmap is not enabled,
- * there are only two trees that will be modified (free space trees), and when
- * rmap is enabled there will be three (freespace + rmap trees). The number of
- * blocks reserved is based on the formula:

fs/xfs/libxfs/xfs_trans_resv.c xfs_calc_buf_res( · line 76

Quoted 10 lines
fs/xfs/libxfs/xfs_trans_resv.c +8 -1
+ * involved in freeing or allocating an extent.
  *
- * num trees * ((2 blocks/level * max depth) - 1)
+ * There are at least two trees that will be modified (free space trees), when
+ * rmap is enabled there will be an additional rmap tree, and when reflinks
+ * are enabled there will be a refcount btree as well
+ *
+ * The number of  blocks reserved is based on the formula:
+ *
+ *     num trees * ((2 blocks/level * max depth) - 1)

Huh, could've sworn I used to have a patch that does this... but I can't find it, so it clearly fell out. Thanks for the patch.

--D

Quoted 15 lines
fs/xfs/libxfs/xfs_trans_resv.c +2 -0
  */
 static uint
 xfs_allocfree_log_count(
@@ -81,6 +84,8 @@ xfs_allocfree_log_count(
 
 	if (xfs_sb_version_hasrmapbt(&mp->m_sb))
 		num_trees++;
+	if (xfs_sb_version_hasreflink(&mp->m_sb))
+		num_trees++;
 
 	return num_ops * num_trees * (2 * mp->m_ag_maxlevels - 1);
 }
Signature · 1 line
2.1.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

I've still seen the assert a couple of times with the fix, and with yesterdays tree update they are back reproducibly (always in generic/168 over NFS).

I suspect the problem is that xfs_refcountbt_alloc_block fully recurses into the allocator and thus could basically double the log reservation.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs