Delayed inode operations not doing the right thing with enospc

I got a lot of these when running stress.sh on my test box

[ 9792.654889] ------------[ cut here ]------------
[ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681
btrfs_alloc_free_block+0xca/0x27c [btrfs]()
[ 9792.654899] Hardware name: To Be Filled By O.E.M.
[ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c

ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]

[ 9792.654919] Pid: 2762, comm: rm Tainted: G        W   2.6.39+ #1
[ 9792.654920] Call Trace:
[ 9792.654922]  [<ffffffff81053c4a>] warn_slowpath_common+0x83/0x9b
[ 9792.654925]  [<ffffffff81053c7c>] warn_slowpath_null+0x1a/0x1c
[ 9792.654933]  [<ffffffffa038e747>] btrfs_alloc_free_block+0xca/0x27c
[btrfs]
[ 9792.654945]  [<ffffffffa03b8562>] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.654953]  [<ffffffffa038189b>] __btrfs_cow_block+0xfc/0x30c [btrfs]
[ 9792.654963]  [<ffffffffa0396aa6>] ? btrfs_buffer_uptodate+0x47/0x58
[btrfs]
[ 9792.654970]  [<ffffffffa0382e48>] ? read_block_for_search+0x94/0x368
[btrfs]
[ 9792.654978]  [<ffffffffa0381ba9>] btrfs_cow_block+0xfe/0x146 [btrfs]
[ 9792.654986]  [<ffffffffa03848b0>] btrfs_search_slot+0x14d/0x4b6 [btrfs]
[ 9792.654997]  [<ffffffffa03b8562>] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.655022]  [<ffffffffa03938e8>] btrfs_lookup_inode+0x2f/0x8f [btrfs]
[ 9792.655025]  [<ffffffff8147afac>] ? _cond_resched+0xe/0x22
[ 9792.655027]  [<ffffffff8147b892>] ? mutex_lock+0x29/0x50
[ 9792.655039]  [<ffffffffa03d41b1>]
btrfs_update_delayed_inode+0x72/0x137 [btrfs]
[ 9792.655051]  [<ffffffffa03d4ea2>] btrfs_run_delayed_items+0x90/0xdb
[btrfs]
[ 9792.655062]  [<ffffffffa039a69b>]
btrfs_commit_transaction+0x228/0x654 [btrfs]
[ 9792.655064]  [<ffffffff8106e8da>] ? remove_wait_queue+0x3a/0x3a
[ 9792.655075]  [<ffffffffa03a2fa5>] btrfs_evict_inode+0x14d/0x202 [btrfs]
[ 9792.655077]  [<ffffffff81132bd6>] evict+0x71/0x111
[ 9792.655079]  [<ffffffff81132de0>] iput+0x12a/0x132
[ 9792.655081]  [<ffffffff8112aa3a>] do_unlinkat+0x106/0x155
[ 9792.655083]  [<ffffffff81127b83>] ? path_put+0x1f/0x23
[ 9792.655085]  [<ffffffff8109c53c>] ? audit_syscall_entry+0x145/0x171
[ 9792.655087]  [<ffffffff81128410>] ? putname+0x34/0x36
[ 9792.655090]  [<ffffffff8112b441>] sys_unlinkat+0x29/0x2b
[ 9792.655092]  [<ffffffff81482c42>] system_call_fastpath+0x16/0x1b
[ 9792.655093] ---[ end trace 02b696eb02b3f768 ]---

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 69 lines · Josef Bacik

I got a lot of these when running stress.sh on my test box

[ 9792.654889] ------------[ cut here ]------------
[ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681
btrfs_alloc_free_block+0xca/0x27c [btrfs]()
[ 9792.654899] Hardware name: To Be Filled By O.E.M.
[ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c

ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]

[ 9792.654919] Pid: 2762, comm: rm Tainted: G        W   2.6.39+ #1
[ 9792.654920] Call Trace:
[ 9792.654922]  [<ffffffff81053c4a>] warn_slowpath_common+0x83/0x9b
[ 9792.654925]  [<ffffffff81053c7c>] warn_slowpath_null+0x1a/0x1c
[ 9792.654933]  [<ffffffffa038e747>] btrfs_alloc_free_block+0xca/0x27c
[btrfs]
[ 9792.654945]  [<ffffffffa03b8562>] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.654953]  [<ffffffffa038189b>] __btrfs_cow_block+0xfc/0x30c [btrfs]
[ 9792.654963]  [<ffffffffa0396aa6>] ? btrfs_buffer_uptodate+0x47/0x58
[btrfs]
[ 9792.654970]  [<ffffffffa0382e48>] ? read_block_for_search+0x94/0x368
[btrfs]
[ 9792.654978]  [<ffffffffa0381ba9>] btrfs_cow_block+0xfe/0x146 [btrfs]
[ 9792.654986]  [<ffffffffa03848b0>] btrfs_search_slot+0x14d/0x4b6 [btrfs]
[ 9792.654997]  [<ffffffffa03b8562>] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.655022]  [<ffffffffa03938e8>] btrfs_lookup_inode+0x2f/0x8f [btrfs]
[ 9792.655025]  [<ffffffff8147afac>] ? _cond_resched+0xe/0x22
[ 9792.655027]  [<ffffffff8147b892>] ? mutex_lock+0x29/0x50
[ 9792.655039]  [<ffffffffa03d41b1>]
btrfs_update_delayed_inode+0x72/0x137 [btrfs]
[ 9792.655051]  [<ffffffffa03d4ea2>] btrfs_run_delayed_items+0x90/0xdb
[btrfs]
[ 9792.655062]  [<ffffffffa039a69b>]
btrfs_commit_transaction+0x228/0x654 [btrfs]
[ 9792.655064]  [<ffffffff8106e8da>] ? remove_wait_queue+0x3a/0x3a
[ 9792.655075]  [<ffffffffa03a2fa5>] btrfs_evict_inode+0x14d/0x202 [btrfs]
[ 9792.655077]  [<ffffffff81132bd6>] evict+0x71/0x111
[ 9792.655079]  [<ffffffff81132de0>] iput+0x12a/0x132
[ 9792.655081]  [<ffffffff8112aa3a>] do_unlinkat+0x106/0x155
[ 9792.655083]  [<ffffffff81127b83>] ? path_put+0x1f/0x23
[ 9792.655085]  [<ffffffff8109c53c>] ? audit_syscall_entry+0x145/0x171
[ 9792.655087]  [<ffffffff81128410>] ? putname+0x34/0x36
[ 9792.655090]  [<ffffffff8112b441>] sys_unlinkat+0x29/0x2b
[ 9792.655092]  [<ffffffff81482c42>] system_call_fastpath+0x16/0x1b
[ 9792.655093] ---[ end trace 02b696eb02b3f768 ]---

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

Quoted 69 lines · Josef Bacik

I got a lot of these when running stress.sh on my test box

[ 9792.654889] ------------[ cut here ]------------
[ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681
btrfs_alloc_free_block+0xca/0x27c [btrfs]()
[ 9792.654899] Hardware name: To Be Filled By O.E.M.
[ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c

ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]

[ 9792.654919] Pid: 2762, comm: rm Tainted: G        W   2.6.39+ #1
[ 9792.654920] Call Trace:
[ 9792.654922]  [<ffffffff81053c4a>] warn_slowpath_common+0x83/0x9b
[ 9792.654925]  [<ffffffff81053c7c>] warn_slowpath_null+0x1a/0x1c
[ 9792.654933]  [<ffffffffa038e747>] btrfs_alloc_free_block+0xca/0x27c
[btrfs]
[ 9792.654945]  [<ffffffffa03b8562>] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.654953]  [<ffffffffa038189b>] __btrfs_cow_block+0xfc/0x30c [btrfs]
[ 9792.654963]  [<ffffffffa0396aa6>] ? btrfs_buffer_uptodate+0x47/0x58
[btrfs]
[ 9792.654970]  [<ffffffffa0382e48>] ? read_block_for_search+0x94/0x368
[btrfs]
[ 9792.654978]  [<ffffffffa0381ba9>] btrfs_cow_block+0xfe/0x146 [btrfs]
[ 9792.654986]  [<ffffffffa03848b0>] btrfs_search_slot+0x14d/0x4b6 [btrfs]
[ 9792.654997]  [<ffffffffa03b8562>] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.655022]  [<ffffffffa03938e8>] btrfs_lookup_inode+0x2f/0x8f [btrfs]
[ 9792.655025]  [<ffffffff8147afac>] ? _cond_resched+0xe/0x22
[ 9792.655027]  [<ffffffff8147b892>] ? mutex_lock+0x29/0x50
[ 9792.655039]  [<ffffffffa03d41b1>]
btrfs_update_delayed_inode+0x72/0x137 [btrfs]
[ 9792.655051]  [<ffffffffa03d4ea2>] btrfs_run_delayed_items+0x90/0xdb
[btrfs]
[ 9792.655062]  [<ffffffffa039a69b>]
btrfs_commit_transaction+0x228/0x654 [btrfs]
[ 9792.655064]  [<ffffffff8106e8da>] ? remove_wait_queue+0x3a/0x3a
[ 9792.655075]  [<ffffffffa03a2fa5>] btrfs_evict_inode+0x14d/0x202 [btrfs]
[ 9792.655077]  [<ffffffff81132bd6>] evict+0x71/0x111
[ 9792.655079]  [<ffffffff81132de0>] iput+0x12a/0x132
[ 9792.655081]  [<ffffffff8112aa3a>] do_unlinkat+0x106/0x155
[ 9792.655083]  [<ffffffff81127b83>] ? path_put+0x1f/0x23
[ 9792.655085]  [<ffffffff8109c53c>] ? audit_syscall_entry+0x145/0x171
[ 9792.655087]  [<ffffffff81128410>] ? putname+0x34/0x36
[ 9792.655090]  [<ffffffff8112b441>] sys_unlinkat+0x29/0x2b
[ 9792.655092]  [<ffffffff81482c42>] system_call_fastpath+0x16/0x1b
[ 9792.655093] ---[ end trace 02b696eb02b3f768 ]---

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

Quoted 22 lines · Josef Bacik

Quoted 21 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete.  This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 9 lines

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please?  Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204.  I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there.  Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow.  Thanks,

Josef

2011/6/7 Josef Bacik <[email protected]>:

Quoted 43 lines · Miao Xie

Quoted 22 lines · Josef Bacik

Quoted 21 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 9 lines

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks,

Quoted 1 line

Josef

Is there a solution for this?

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv:

2011/6/7 Josef Bacik <[email protected]>:

Quoted 43 lines · Miao Xie

Quoted 22 lines · Josef Bacik

Quoted 21 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 9 lines

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks,

Quoted 1 line

Josef

Is there a solution for this?

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv:

Quoted 9 lines

2011/6/7 Josef Bacik <[email protected]>:

Quoted 8 lines · Miao Xie

Quoted 7 lines · Josef Bacik

Quoted 6 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should hav=

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

reserved enough space for those operations to complete. =A0This i=

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

happening because use_block_rsv() will call get_block_rsv(), whic=

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

root->ref_cows is set (which is the case on all fs roots) we will=

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

trans->block_rsv, which will only have what the current transacti=

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

starter had reserved.

What needs to be done instead is we need to have a block reserve =

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

any reservation that is done at create time for these inodes is m=

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

to this special reserve, and then when you run the delayed inode =

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

stuff you set trans->block_rsv to the special block reserve so th=

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

accounting is all done properly.

This is just off the top of my head, there may be a better way to=

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

I've not actually looked that the delayed inode code at all.

I would do this myself but I have a ever increasing list of shit =

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 1 line

so will somebody pick this up and fix it please? =A0Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we hav=

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204. =A0I tried mak=

Quoted 1 line

Quoted 1 line

reserve_metadata_bytes commit the transaction regardless of whether=

Quoted 1 line

Quoted 1 line

not there were pinned bytes but the test just hung there. =A0Usuall=

Quoted 1 line

Quoted 1 line

takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes=

Quoted 2 lines

Quoted 2 lines

204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we=

Quoted 1 line

Quoted 1 line

reclaim the space they are holding onto so we don't get enospc, and=

Quoted 1 line

Quoted 1 line

needs to be better than just committing the transaction because tha=

Quoted 6 lines

Quoted 3 lines

dog slow. =A0Thanks,

Josef

Is there a solution for this?

Quoted 5 lines

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv:

Ok I think I've got this nailed down. =A0Will you run with this patch=

2011/7/13 Josef Bacik <[email protected]>:

Quoted 55 lines · Christian Brunner

Quoted 53 lines

2011/6/7 Josef Bacik <[email protected]>:

Quoted 43 lines · Miao Xie

Quoted 22 lines · Josef Bacik

Quoted 21 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 9 lines

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks,

Quoted 1 line

Josef

Is there a solution for this?

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv:

Quoted 1 line

Ok I think I've got this nailed down. Will you run with this patch and make sure the warnings go away? Thanks,

I'm sorry, I'm still getting a lot of warnings like the one below.

I've also noticed, that I'm not getting these messages when the free_space_cache is disabled.

2011/7/13 Josef Bacik <[email protected]>:

Quoted 55 lines · Christian Brunner

Quoted 53 lines

2011/6/7 Josef Bacik <[email protected]>:

Quoted 43 lines · Miao Xie

Quoted 22 lines · Josef Bacik

Quoted 21 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 9 lines

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks,

Quoted 1 line

Josef

Is there a solution for this?

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv:

Quoted 1 line

Ok I think I've got this nailed down. Will you run with this patch and make sure the warnings go away? Thanks,

I'm sorry, I'm still getting a lot of warnings like the one below.

I've also noticed, that I'm not getting these messages when the free_space_cache is disabled.

2011/7/13 Josef Bacik <[email protected]>:

Quoted 55 lines · Christian Brunner

Quoted 53 lines

2011/6/7 Josef Bacik <[email protected]>:

Quoted 43 lines · Miao Xie

Quoted 22 lines · Josef Bacik

Quoted 21 lines

I got a lot of these when running stress.sh on my test box

This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root->ref_cows is set (which is the case on all fs roots) we will use trans->block_rsv, which will only have what the current transaction starter had reserved.

What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans->block_rsv to the special block reserve so the accounting is all done properly.

This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all.

Quoted 9 lines

Quoted 2 lines

I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks,

Sorry, it's my miss.
I forgot to set trans->block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks,

Quoted 1 line

Josef

Is there a solution for this?

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv:

Quoted 1 line

Ok I think I've got this nailed down. Will you run with this patch and make sure the warnings go away? Thanks,

I'm sorry, I'm still getting a lot of warnings like the one below.

I've also noticed, that I'm not getting these messages when the free_space_cache is disabled.