PLEASE TEST: Everybody who is seeing weird and long hangs

PATCH Josef Bacik 3 participants 8 messages Aug 1, 2011 – Aug 2, 2011 Btrfs file system

Share & tools

Josef Bacik

author

2011-08-01 15:21 UTC #0 lore raw

Hello,

We've seen a lot of reports of people having these constant long pauses when doing things like sync or such. The stack traces usually all look the same, one is btrfs-transaction stuck in btrfs_wait_marked_extents and one is btrfs-submit-# stuck in get_request_wait. I had originally thought this was due to the new plugging stuff, but I think it just makes the problem happen more quickly as we've seen that 2.6.38 which we thought was ok will still have the problem happen if given enough time.

I _think_ this is because of the way we write out metadata in the transaction commit phase. We're doing write_on_page for every dirty page in the btree during the commit. This sucks because basically we end up with one bio per page, which makes us blow out our nr_requests constantly, which is why btrfs-submit-# is always stuck in get_request_wait. What we need to do instead is use filemap_fdatawrite which will do a WB_SYNC_ALL but will do it via writepages, so hopefully we will get less bios and this problem will go away. Please try this very hastily put together patch if you are experiencing this problem and let me know if it fixes it for you. Thanks,

Josef

fs/btrfs/transaction.c +0 -0

index eb55863..86217a4 100644
@@ -577,15 +577,17 @@ int btrfs_end_transaction_dmeta(struct

btrfs_trans_handle *trans,
int btrfs_write_marked_extents(struct btrfs_root *root,
struct extent_io_tree *dirty_pages, int mark)
{
- int ret;
- int err = 0;
- int werr = 0;
- struct page *page;
+// int ret;
+// int err = 0;
+// int werr = 0;
+// struct page *page;
struct inode *btree_inode = root->fs_info->btree_inode;
- u64 start = 0;
- u64 end;
- unsigned long index;
+// u64 start = 0;
+// u64 end;
+// unsigned long index;

+ return filemap_fdatawrite(btree_inode->i_mapping);
+ /*
while (1) {
ret = find_first_extent_bit(dirty_pages, start, &start, &end,
mark);
@@ -624,7 +626,8 @@ int btrfs_write_marked_extents(struct btrfs_root *root,
}
if (err)
werr = err;
- return werr;
+ */
+// return werr;
}

Chris Mason

2011-08-01 15:45 UTC #1 lore raw

Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -0400:

Quoted 10 lines

Hello,
We've seen a lot of reports of people having these constant long pauses when doing things like sync or such. The stack traces usually all look the same, one is btrfs-transaction stuck in btrfs_wait_marked_extents and one is btrfs-submit-# stuck in get_request_wait. I had originally thought this was due to the new plugging stuff, but I think it just makes the problem happen more quickly as we've seen that 2.6.38 which we thought was ok will still have the problem happen if given enough time.

Quoted 10 lines

I _think_ this is because of the way we write out metadata in the transaction commit phase. We're doing write_on_page for every dirty page in the btree during the commit. This sucks because basically we end up with one bio per page, which makes us blow out our nr_requests constantly, which is why btrfs-submit-# is always stuck in get_request_wait. What we need to do instead is use filemap_fdatawrite which will do a WB_SYNC_ALL but will do it via writepages, so hopefully we will get less bios and this problem will go away. Please try this very hastily put together patch if you are experiencing this problem and let me know if it fixes it for you. Thanks,

I'm definitely curious to hear if this helps, but I think it might cause a different set of problems. It writes everything that is dirty on the btree, which includes a lot of things we've cow'd in the current transaction and marked dirty. They will have to go through COW again if someone wants to modify them again.

The btrfs writepage code does this:

        ret = __extent_writepage(page, wbc, &epd);

        extent_write_cache_pages(tree, mapping, &wbc_writepages,
                                 __extent_writepage, &epd, flush_write_bio);
        flush_epd_write_bio(&epd);

So during the commit phase we'll be grabbing adjacent pages already. My bet is that our problem will go away if we remove this extra IO completely.

-chris

Quoted 43 lines

Josef

fs/btrfs/transaction.c +9 -7

index eb55863..86217a4 100644
@@ -577,15 +577,17 @@ int btrfs_end_transaction_dmeta(struct
 btrfs_trans_handle *trans,
 int btrfs_write_marked_extents(struct btrfs_root *root,
                    struct extent_io_tree *dirty_pages, int mark)
 {
-    int ret;
-    int err = 0;
-    int werr = 0;
-    struct page *page;
+//    int ret;
+//    int err = 0;
+//    int werr = 0;
+//    struct page *page;
     struct inode *btree_inode = root->fs_info->btree_inode;
-    u64 start = 0;
-    u64 end;
-    unsigned long index;
+//    u64 start = 0;
+//    u64 end;
+//    unsigned long index;
 
+    return filemap_fdatawrite(btree_inode->i_mapping);
+    /*
     while (1) {
         ret = find_first_extent_bit(dirty_pages, start, &start, &end,

                         mark);

fs/btrfs/transaction.c +2 -1

@@ -624,7 +626,8 @@ int btrfs_write_marked_extents(struct btrfs_root *root,
     }
     if (err)
         werr = err;
-    return werr;
+    */
+//    return werr;
 }
 
 /*

Josef Bacik

author

2011-08-01 16:03 UTC #2 lore raw

Quoted 22 lines · Chris Mason

Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -0400:
Quoted 10 lines
Hello,
We've seen a lot of reports of people having these constant long pauses when doing things like sync or such. The stack traces usually all look the same, one is btrfs-transaction stuck in btrfs_wait_marked_extents and one is btrfs-submit-# stuck in get_request_wait. I had originally thought this was due to the new plugging stuff, but I think it just makes the problem happen more quickly as we've seen that 2.6.38 which we thought was ok will still have the problem happen if given enough time.
Quoted 10 lines
I _think_ this is because of the way we write out metadata in the transaction commit phase. We're doing write_on_page for every dirty page in the btree during the commit. This sucks because basically we end up with one bio per page, which makes us blow out our nr_requests constantly, which is why btrfs-submit-# is always stuck in get_request_wait. What we need to do instead is use filemap_fdatawrite which will do a WB_SYNC_ALL but will do it via writepages, so hopefully we will get less bios and this problem will go away. Please try this very hastily put together patch if you are experiencing this problem and let me know if it fixes it for you. Thanks,

Quoted 6 lines

I'm definitely curious to hear if this helps, but I think it might cause a different set of problems. It writes everything that is dirty on the btree, which includes a lot of things we've cow'd in the current transaction and marked dirty. They will have to go through COW again if someone wants to modify them again.

But this is happening in the commit after we've done all of our work, we shouldn't be dirtying anything else at this point right?

Quoted 8 lines

The btrfs writepage code does this:

        ret = __extent_writepage(page, wbc, &epd);

        extent_write_cache_pages(tree, mapping, &wbc_writepages,
                                 __extent_writepage, &epd, flush_write_bio);
        flush_epd_write_bio(&epd);

Yeah but nr_to_write is 1, so after the __extent_writepage it will be 0 and extent_write_cache_pages will just return since there's nothing to write, so we'll still end up with 1 page at a time being written out. Thanks,

Josef

Chris Mason

2011-08-01 17:54 UTC #3 lore raw

Excerpts from Josef Bacik's message of 2011-08-01 12:03:34 -0400:

Quoted 30 lines · Chris Mason

Quoted 22 lines
Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -0400:
Quoted 10 lines
Hello,
We've seen a lot of reports of people having these constant long pauses when doing things like sync or such. The stack traces usually all look the same, one is btrfs-transaction stuck in btrfs_wait_marked_extents and one is btrfs-submit-# stuck in get_request_wait. I had originally thought this was due to the new plugging stuff, but I think it just makes the problem happen more quickly as we've seen that 2.6.38 which we thought was ok will still have the problem happen if given enough time.
Quoted 10 lines
I _think_ this is because of the way we write out metadata in the transaction commit phase. We're doing write_on_page for every dirty page in the btree during the commit. This sucks because basically we end up with one bio per page, which makes us blow out our nr_requests constantly, which is why btrfs-submit-# is always stuck in get_request_wait. What we need to do instead is use filemap_fdatawrite which will do a WB_SYNC_ALL but will do it via writepages, so hopefully we will get less bios and this problem will go away. Please try this very hastily put together patch if you are experiencing this problem and let me know if it fixes it for you. Thanks,
Quoted 6 lines
I'm definitely curious to hear if this helps, but I think it might cause a different set of problems. It writes everything that is dirty on the btree, which includes a lot of things we've cow'd in the current transaction and marked dirty. They will have to go through COW again if someone wants to modify them again.

Quoted 2 lines

But this is happening in the commit after we've done all of our work, we shouldn't be dirtying anything else at this point right?

The commit code is setup to unblock people before we start the IO:

       trans->transaction->blocked = 0;
        spin_lock(&root->fs_info->trans_lock);
        root->fs_info->running_transaction = NULL;
        root->fs_info->trans_no_join = 0;
        spin_unlock(&root->fs_info->trans_lock);
        mutex_unlock(&root->fs_info->reloc_mutex);
        
        wake_up(&root->fs_info->transaction_wait);

        ret = btrfs_write_and_wait_transaction(trans, root);

So, we should have concurrent FS mods for a new transaction while we are writing out this old transaction.

Quoted 10 lines

Quoted 9 lines

The btrfs writepage code does this:

        ret = __extent_writepage(page, wbc, &epd);

        extent_write_cache_pages(tree, mapping, &wbc_writepages,
                                 __extent_writepage, &epd, flush_write_bio);
        flush_epd_write_bio(&epd);

Quoted 4 lines

Yeah but nr_to_write is 1, so after the __extent_writepage it will be 0 and extent_write_cache_pages will just return since there's nothing to write, so we'll still end up with 1 page at a time being written out. Thanks,

We bump nr_to_write to 64:

	struct writeback_control wbc_writepages = {
		.sync_mode	= wbc->sync_mode,
		.older_than_this = NULL,
		.nr_to_write	= 64,
		.range_start	= page_offset(page) + PAGE_CACHE_SIZE,
		.range_end	= (loff_t)-1,
	};

	ret = __extent_writepage(page, wbc, &epd);

	extent_write_cache_pages(tree, mapping, &wbc_writepages,
				 __extent_writepage, &epd, flush_write_bio);
	flush_epd_write_bio(&epd);

-chris

Josef Bacik

author

2011-08-01 18:01 UTC #4 lore raw

Quoted 47 lines · Chris Mason

Excerpts from Josef Bacik's message of 2011-08-01 12:03:34 -0400:
Quoted 30 lines · Chris Mason
Quoted 22 lines
Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -0400:
Quoted 10 lines
Hello,
We've seen a lot of reports of people having these constant long pauses when doing things like sync or such. The stack traces usually all look the same, one is btrfs-transaction stuck in btrfs_wait_marked_extents and one is btrfs-submit-# stuck in get_request_wait. I had originally thought this was due to the new plugging stuff, but I think it just makes the problem happen more quickly as we've seen that 2.6.38 which we thought was ok will still have the problem happen if given enough time.
Quoted 10 lines
I _think_ this is because of the way we write out metadata in the transaction commit phase. We're doing write_on_page for every dirty page in the btree during the commit. This sucks because basically we end up with one bio per page, which makes us blow out our nr_requests constantly, which is why btrfs-submit-# is always stuck in get_request_wait. What we need to do instead is use filemap_fdatawrite which will do a WB_SYNC_ALL but will do it via writepages, so hopefully we will get less bios and this problem will go away. Please try this very hastily put together patch if you are experiencing this problem and let me know if it fixes it for you. Thanks,
Quoted 6 lines
I'm definitely curious to hear if this helps, but I think it might cause a different set of problems. It writes everything that is dirty on the btree, which includes a lot of things we've cow'd in the current transaction and marked dirty. They will have to go through COW again if someone wants to modify them again.
Quoted 2 lines
But this is happening in the commit after we've done all of our work, we shouldn't be dirtying anything else at this point right?
The commit code is setup to unblock people before we start the IO:
       trans->transaction->blocked = 0;
        spin_lock(&root->fs_info->trans_lock);
        root->fs_info->running_transaction = NULL;
        root->fs_info->trans_no_join = 0;
        spin_unlock(&root->fs_info->trans_lock);
        mutex_unlock(&root->fs_info->reloc_mutex);
        
        wake_up(&root->fs_info->transaction_wait);

        ret = btrfs_write_and_wait_transaction(trans, root);

Quoted 3 lines

So, we should have concurrent FS mods for a new transaction while we are writing out this old transaction.

Ah right, but then this brings up another question, we shouldn't cow them again since we would have set the new transid. And isn't this kind of bad, since somebody could come in and dirty a piece of metadata before we have a chance to write it out for this transaction, so we end up writing out the new data instead of what we are trying to commit?

And also the writepages() thing would get around this problem since we are SYNC_ALL which now tags all dirty pages as TOWRITE and then writes those pages instead of writing all dirty pages. So anything being dirtied once we started writepages would be fine.

So this really could explain why this is sucking for people, we are just walking through and writing everything that's dirty, and then doing the same thing in wait_marked_extents() again, so we could be writing out things that aren't in the transaction that we committed, which would mean we're writing way more than we need to.

Quoted 17 lines

Quoted 10 lines
Quoted 9 lines
The btrfs writepage code does this:
        ret = __extent_writepage(page, wbc, &epd);

        extent_write_cache_pages(tree, mapping, &wbc_writepages,
                                 __extent_writepage, &epd, flush_write_bio);
        flush_epd_write_bio(&epd);
Quoted 4 lines
Yeah but nr_to_write is 1, so after the __extent_writepage it will be 0 and extent_write_cache_pages will just return since there's nothing to write, so we'll still end up with 1 page at a time being written out. Thanks,
We bump nr_to_write to 64:

Quoted 14 lines

	struct writeback_control wbc_writepages = {
		.sync_mode	= wbc->sync_mode,
		.older_than_this = NULL,
		.nr_to_write	= 64,
		.range_start	= page_offset(page) + PAGE_CACHE_SIZE,
		.range_end	= (loff_t)-1,
	};

	ret = __extent_writepage(page, wbc, &epd);

	extent_write_cache_pages(tree, mapping, &wbc_writepages,
				 __extent_writepage, &epd, flush_write_bio);
	flush_epd_write_bio(&epd);

Oops I missed that, thanks,

Josef

Chris Mason

2011-08-01 18:21 UTC #5 lore raw

Excerpts from Josef Bacik's message of 2011-08-01 14:01:35 -0400:

Quoted 52 lines · Chris Mason

Quoted 47 lines
Excerpts from Josef Bacik's message of 2011-08-01 12:03:34 -0400:
Quoted 30 lines · Chris Mason
Quoted 22 lines
Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -0400:
Quoted 10 lines
Hello,
We've seen a lot of reports of people having these constant long pauses when doing things like sync or such. The stack traces usually all look the same, one is btrfs-transaction stuck in btrfs_wait_marked_extents and one is btrfs-submit-# stuck in get_request_wait. I had originally thought this was due to the new plugging stuff, but I think it just makes the problem happen more quickly as we've seen that 2.6.38 which we thought was ok will still have the problem happen if given enough time.
Quoted 10 lines
I _think_ this is because of the way we write out metadata in the transaction commit phase. We're doing write_on_page for every dirty page in the btree during the commit. This sucks because basically we end up with one bio per page, which makes us blow out our nr_requests constantly, which is why btrfs-submit-# is always stuck in get_request_wait. What we need to do instead is use filemap_fdatawrite which will do a WB_SYNC_ALL but will do it via writepages, so hopefully we will get less bios and this problem will go away. Please try this very hastily put together patch if you are experiencing this problem and let me know if it fixes it for you. Thanks,
Quoted 6 lines
I'm definitely curious to hear if this helps, but I think it might cause a different set of problems. It writes everything that is dirty on the btree, which includes a lot of things we've cow'd in the current transaction and marked dirty. They will have to go through COW again if someone wants to modify them again.
Quoted 2 lines
But this is happening in the commit after we've done all of our work, we shouldn't be dirtying anything else at this point right?
The commit code is setup to unblock people before we start the IO:
       trans->transaction->blocked = 0;
        spin_lock(&root->fs_info->trans_lock);
        root->fs_info->running_transaction = NULL;
        root->fs_info->trans_no_join = 0;
        spin_unlock(&root->fs_info->trans_lock);
        mutex_unlock(&root->fs_info->reloc_mutex);
        
        wake_up(&root->fs_info->transaction_wait);

        ret = btrfs_write_and_wait_transaction(trans, root);
Quoted 3 lines
So, we should have concurrent FS mods for a new transaction while we are writing out this old transaction.

Quoted 5 lines

Ah right, but then this brings up another question, we shouldn't cow them again since we would have set the new transid. And isn't this kind of bad, since somebody could come in and dirty a piece of metadata before we have a chance to write it out for this transaction, so we end up writing out the new data instead of what we are trying to commit?

I think we're mixing together different ideas here. If we're doing a commit on transaction N, we allow N+1 to start while we're doing the btrfs_write_and_wait_transaction(). N+1 might allocate and dirty a new block, which btrfs_write_and_wait_transaction might start IO on.

Strictly speaking this isn't a problem. It doesn't break any rules of COW because we're allowed to write metadata at any time. But, once we do write it, we must COW it again if we want to change it. So, anything that btrfs_write_and_wait_transaction() catches from transaction N+1 is likely to make more work for us because future mods will have to allocate a new block. Basically it's wasted IO.

But, it's also free IO, assuming it was contiguous. The problem is that write_cache_pages isn't actually making sure it was contiguous, so we end up doing many more writes than we could have.

-chris

cwillu

2011-08-01 23:28 UTC #6 lore raw

On Mon, Aug 1, 2011 at 12:21 PM, Chris Mason <[email protected]> w= rote:

Quoted 5 lines

Excerpts from Josef Bacik's message of 2011-08-01 14:01:35 -0400:
Quoted 4 lines · Chris Mason
Quoted 3 lines
Excerpts from Josef Bacik's message of 2011-08-01 12:03:34 -0400:
Quoted 2 lines · Chris Mason
Quoted 1 line
Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -0400=

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hello,
We've seen a lot of reports of people having these constant lon=

g pauses

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
when doing things like sync or such. =C2=A0The stack traces usu=

ally all look

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the same, one is btrfs-transaction stuck in btrfs_wait_marked_e=

xtents

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and one is btrfs-submit-# stuck in get_request_wait. =C2=A0I ha=

d originally

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thought this was due to the new plugging stuff, but I think it =

just

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
makes the problem happen more quickly as we've seen that 2.6.38=

which we

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thought was ok will still have the problem happen if given enou=

gh time.

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I _think_ this is because of the way we write out metadata in t=

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
transaction commit phase. =C2=A0We're doing write_on_page for e=

very dirty

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
page in the btree during the commit. =C2=A0This sucks because b=

asically we

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
end up with one bio per page, which makes us blow out our nr_re=

quests

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
constantly, which is why btrfs-submit-# is always stuck in get_request_wait. =C2=A0What we need to do instead is use filem=

ap_fdatawrite

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
which will do a WB_SYNC_ALL but will do it via writepages, so h=

opefully

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will get less bios and this problem will go away. =C2=A0Plea=

se try this

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
very hastily put together patch if you are experiencing this pr=

oblem and

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
let me know if it fixes it for you. =C2=A0Thanks,
I'm definitely curious to hear if this helps, but I think it mig=

ht cause

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
a different set of problems. =C2=A0It writes everything that is =

dirty on the

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
btree, which includes a lot of things we've cow'd in the current transaction and marked dirty. =C2=A0They will have to go through=

COW again

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
if someone wants to modify them again.
But this is happening in the commit after we've done all of our w=

ork, we

Quoted 7 lines

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
shouldn't be dirtying anything else at this point right?
The commit code is setup to unblock people before we start the IO:
=C2=A0 =C2=A0 =C2=A0 =C2=A0trans->transaction->blocked =3D 0; =C2=A0 =C2=A0 =C2=A0 =C2=A0 spin_lock(&root->fs_info->trans_lock); =C2=A0 =C2=A0 =C2=A0 =C2=A0 root->fs_info->running_transaction =3D=

NULL;

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0 root->fs_info->trans_no_join =3D 0; =C2=A0 =C2=A0 =C2=A0 =C2=A0 spin_unlock(&root->fs_info->trans_lock=

);

Quoted 1 line

Quoted 1 line
Quoted 1 line
=C2=A0 =C2=A0 =C2=A0 =C2=A0 mutex_unlock(&root->fs_info->reloc_mut=

ex);

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0 wake_up(&root->fs_info->transaction_wa=

it);

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ret =3D btrfs_write_and_wait_transacti=

on(trans, root);

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
So, we should have concurrent FS mods for a new transaction while =

we are

Quoted 5 lines

Quoted 5 lines
Quoted 2 lines
writing out this old transaction.
Ah right, but then this brings up another question, we shouldn't cow them again since we would have set the new transid. =C2=A0And isn't =

this kind

Quoted 2 lines

Quoted 2 lines
of bad, since somebody could come in and dirty a piece of metadata before we have a chance to write it out for this transaction, so we =

end

Quoted 3 lines

Quoted 1 line
up writing out the new data instead of what we are trying to commit?
I think we're mixing together different ideas here. =C2=A0If we're do=

ing a

Quoted 2 lines

commit on transaction N, we allow N+1 to start while we're doing the btrfs_write_and_wait_transaction(). =C2=A0N+1 might allocate and dirt=

y a new

Quoted 3 lines

block, which btrfs_write_and_wait_transaction might start IO on.
Strictly speaking this isn't a problem. =C2=A0It doesn't break any ru=

les of

Quoted 1 line

COW because we're allowed to write metadata at any time. =C2=A0But, o=

nce we

Quoted 1 line

do write it, we must COW it again if we want to change it. =C2=A0So, =

anything

Quoted 1 line

that btrfs_write_and_wait_transaction() catches from transaction N+1 =

Quoted 4 lines

likely to make more work for us because future mods will have to allocate a new block. =C2=A0Basically it's wasted IO.
But, it's also free IO, assuming it was contiguous. =C2=A0The problem=

is that

Quoted 6 lines

write_cache_pages isn't actually making sure it was contiguous, so we end up doing many more writes than we could have.
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs=

" in

Quoted 2 lines

the body of a message to [email protected]
More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info.ht=

Quoted 1 line

=46irst user ("youagree") reported back on irc:

<youagree> guys, just came to report its much worse with josef's patch <youagree> now i can hardly start anything, it's slowed down most of th= e time
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Chris Mason

2011-08-02 00:09 UTC #7 lore raw

Excerpts from cwillu's message of 2011-08-01 19:28:35 -0400:

Quoted 1 line

On Mon, Aug 1, 2011 at 12:21 PM, Chris Mason <[email protected]>=

Quoted 3 lines · wrote:

Quoted 3 lines
Excerpts from Josef Bacik's message of 2011-08-01 14:01:35 -0400:
Quoted 2 lines · Chris Mason
Quoted 1 line
Excerpts from Josef Bacik's message of 2011-08-01 12:03:34 -0400=

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Chris Mason
Quoted 1 line
Excerpts from Josef Bacik's message of 2011-08-01 11:21:34 -04=

00:

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hello,
We've seen a lot of reports of people having these constant l=

ong pauses

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
when doing things like sync or such. =C2=A0The stack traces u=

sually all look

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the same, one is btrfs-transaction stuck in btrfs_wait_marked=

_extents

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and one is btrfs-submit-# stuck in get_request_wait. =C2=A0I =

had originally

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thought this was due to the new plugging stuff, but I think i=

t just

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
makes the problem happen more quickly as we've seen that 2.6.=

38 which we

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thought was ok will still have the problem happen if given en=

ough time.

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I _think_ this is because of the way we write out metadata in=

the

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
transaction commit phase. =C2=A0We're doing write_on_page for=

every dirty

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
page in the btree during the commit. =C2=A0This sucks because=

basically we

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
end up with one bio per page, which makes us blow out our nr_=

requests

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
constantly, which is why btrfs-submit-# is always stuck in get_request_wait. =C2=A0What we need to do instead is use fil=

emap_fdatawrite

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
which will do a WB_SYNC_ALL but will do it via writepages, so=

hopefully

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
we will get less bios and this problem will go away. =C2=A0Pl=

ease try this

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
very hastily put together patch if you are experiencing this =

problem and

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
let me know if it fixes it for you. =C2=A0Thanks,
I'm definitely curious to hear if this helps, but I think it m=

ight cause

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
a different set of problems. =C2=A0It writes everything that i=

s dirty on the

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
btree, which includes a lot of things we've cow'd in the curre=

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
transaction and marked dirty. =C2=A0They will have to go throu=

gh COW again

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
if someone wants to modify them again.
But this is happening in the commit after we've done all of our=

work, we

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
shouldn't be dirtying anything else at this point right?
The commit code is setup to unblock people before we start the I=

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0trans->transaction->blocked =3D 0; =C2=A0 =C2=A0 =C2=A0 =C2=A0 spin_lock(&root->fs_info->trans_lock=

);

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
=C2=A0 =C2=A0 =C2=A0 =C2=A0 root->fs_info->running_transaction =3D=

NULL;

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0 root->fs_info->trans_no_join =3D 0; =C2=A0 =C2=A0 =C2=A0 =C2=A0 spin_unlock(&root->fs_info->trans_lo=

ck);

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
=C2=A0 =C2=A0 =C2=A0 =C2=A0 mutex_unlock(&root->fs_info->reloc_m=

utex);

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0 wake_up(&root->fs_info->transaction_=

wait);

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ret =3D btrfs_write_and_wait_transac=

tion(trans, root);

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
So, we should have concurrent FS mods for a new transaction whil=

e we are

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
writing out this old transaction.
Ah right, but then this brings up another question, we shouldn't c=

Quoted 1 line

Quoted 1 line
Quoted 1 line
them again since we would have set the new transid. =C2=A0And isn'=

t this kind

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
of bad, since somebody could come in and dirty a piece of metadata before we have a chance to write it out for this transaction, so w=

e end

Quoted 1 line

Quoted 1 line
Quoted 1 line
up writing out the new data instead of what we are trying to commi=

Quoted 2 lines

Quoted 2 lines
I think we're mixing together different ideas here. =C2=A0If we're =

doing a

Quoted 1 line

Quoted 1 line
commit on transaction N, we allow N+1 to start while we're doing th=

Quoted 1 line

Quoted 1 line
btrfs_write_and_wait_transaction(). =C2=A0N+1 might allocate and di=

rty a new

Quoted 3 lines

Quoted 3 lines
block, which btrfs_write_and_wait_transaction might start IO on.
Strictly speaking this isn't a problem. =C2=A0It doesn't break any =

rules of

Quoted 1 line

Quoted 1 line
COW because we're allowed to write metadata at any time. =C2=A0But,=

once we

Quoted 1 line

Quoted 1 line
do write it, we must COW it again if we want to change it. =C2=A0So=

, anything

Quoted 1 line

Quoted 1 line
that btrfs_write_and_wait_transaction() catches from transaction N+=

1 is

Quoted 4 lines

Quoted 4 lines
likely to make more work for us because future mods will have to allocate a new block. =C2=A0Basically it's wasted IO.
But, it's also free IO, assuming it was contiguous. =C2=A0The probl=

em is that

Quoted 1 line

Quoted 1 line
write_cache_pages isn't actually making sure it was contiguous, so =

Quoted 5 lines

Quoted 1 line
end up doing many more writes than we could have.
=20
First user ("youagree") reported back on irc:
=20
<youagree> guys, just came to report its much worse with josef's patc=

Quoted 1 line

<youagree> now i can hardly start anything, it's slowed down most of =

the time

Josef's filemap_fdatawrite patch? He sent a second one to the list tha= t
gets rid of the extra IO done by the current code. That's the one we hope will fix things.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html