delalloc and reservation.

DISCUSSION Aneesh Kumar K.V 2 participants 9 messages Oct 29, 2007 – Oct 30, 2007 ext4 file system

Share & tools

Aneesh Kumar K.V

author

2007-10-29 10:01 UTC #0 lore raw

Hi All,

I looked at the delalloc and reservation differences that Valerie was observing.
Below is my understanding. I am not sure whether the below will result in higher fragmentation that Eric Sandeen is observing. I guess it should not. Even though the reservation gets discarded during the clear inode due to memory pressure
the request for new reservation should get the blocks nearby and not break extents right ?

any how below is the simple case.

without delalloc the blocks are requested during prepare_write/write_begin. That means we enter ext4_new_blocks_old which will call ext4_try_to_allocate_with_rsv. Now if there is no reservation for this inode a new one will be allocated. After using the blocks this reservation is destroyed during the close via ext4_release_file

With delalloc the blocks are not requested until we hit writeback/ext4_da_writepages That means if we create new file and close them the reservation will be discarded during close via ext4_release_file.( Actually there will be nothing to clear) Now when we do a sync/or write back. We try to get the block, the inode will request for new reservation. This reservation is not discarded untill we call clear_inode and that results in the behavior we are seeing.

Free blocks: 1440-8191, 8194-8199, 8202-8207, 8210-8215, 8218-8223, 8226-8231, 8234-8239, 8242-8247, 8250-8255, 8258-8263, 8266-8271, 8274-8279, 8282-8287, 8290-8295, 8298-8303, 8306-8311, 8314-8319, 8322-8327, 8330-8335, 8338-8343, 8346-12799

So now the question is where do we discard the reservation in case of delalloc.

-aneesh

Alex Tomas

2007-10-29 14:26 UTC #1 lore raw

Hi,

could you try the patch attached. it should fix the issue. the idea was to align requests in order to help raid5-like setups. but somewhere I lost one bit in mballoc: it should pre-allocate all crossed stripes, but it didn't.

as for discard, lustre doesn't use open/close for data, so discard-on-close makes zero sense in our case. I'm not very positive whether we need to drop preallocation on file close in case of delayed allocation as writeback can be started while file is open and finish after close(2).

thanks, Alex

Quoted 27 lines · Aneesh Kumar K.V

Hi All,
I looked at the delalloc and reservation differences that Valerie was observing.
Below is my understanding. I am not sure whether the below will result in higher fragmentation that Eric Sandeen is observing. I guess it should not. Even though the reservation gets discarded during the clear inode due to memory pressure
the request for new reservation should get the blocks nearby and not break extents right ?
any how below is the simple case.
without delalloc the blocks are requested during prepare_write/write_begin. That means we enter ext4_new_blocks_old which will call ext4_try_to_allocate_with_rsv. Now if there is no reservation for this inode a new one will be allocated. After using the blocks this reservation is destroyed during the close via ext4_release_file
With delalloc the blocks are not requested until we hit writeback/ext4_da_writepages
That means if we create new file and close them the reservation will be discarded
during close via ext4_release_file.( Actually there will be nothing to clear)
Now when we do a sync/or write back. We try to get the block, the inode will
request for new reservation. This reservation is not discarded untill we call clear_inode
and that results in the behavior we are seeing.
Free blocks: 1440-8191, 8194-8199, 8202-8207, 8210-8215, 8218-8223, 8226-8231, 8234-8239, 8242-8247, 8250-8255, 8258-8263, 8266-8271, 8274-8279, 8282-8287, 8290-8295, 8298-8303, 8306-8311, 8314-8319, 8322-8327, 8330-8335, 8338-8343, 8346-12799
So now the question is where do we discard the reservation in case of delalloc.
(quoted diff) +0 -2
-aneesh
-

Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

fs/ext4/mballoc.c +7 -2

Index: linux-2.6.24-rc1/fs/ext4/mballoc.c
===================================================================
@@ -3088,8 +3088,10 @@ static void ext4_mb_normalize_request(st
 			break;
 		}
 	}
+	size = wind;
+
 	if (wind == 0) {
-		__u64 tstart;
+		__u64 tstart, tend;
 		/* file is quite large, we now preallocate with
 		 * the biggest configured window with regart to
 		 * logical offset */

@@ -3097,8 +3099,11 @@ static void ext4_mb_normalize_request(st
 		tstart = ac->ac_o_ex.fe_logical;
 		do_div(tstart, wind);
 		start = tstart * wind;
+		tend = ac->ac_o_ex.fe_logical + ac->ac_o_ex.fe_len - 1;
+		do_div(tend, wind);
+		tend = tend * wind + wind;
+		size = tend - start;
 	}
-	size = wind;
 	orig_size = size;
 	orig_start = start;

Aneesh Kumar K.V

author

2007-10-29 15:05 UTC #2 lore raw

Quoted 7 lines · Alex Tomas

Hi,
could you try the patch attached. it should fix the issue. the idea was to align requests in order to help raid5-like setups. but somewhere I lost one bit in mballoc: it should pre-allocate all crossed stripes, but it didn't.

Quoted 6 lines

as for discard, lustre doesn't use open/close for data, so discard-on-close makes zero sense in our case. I'm not very positive whether we need to drop preallocation on file close in case of delayed allocation as writeback can be started while file is open and finish after close(2).

mballoc by default doesn't give the particular layout only if i force small size to use inode preallocation i am hitting the problem. ie to change the below line in ext4_mb_group_or_file

if (ac->ac_o_ex.fe_len >= sbi->s_mb_small_req)

if (ac->ac_o_ex.fe_len <= sbi->s_mb_small_req)

Do you want to test the patch with this change ?

We are observing the problem with delalloc and nomballoc.

-aneesh

Alex Tomas

2007-10-29 14:45 UTC #3 lore raw

sorry, I don't quite understand how do you observe this with nomballoc

thanks, Alex

Quoted 16 lines · Aneesh Kumar K.V

mballoc by default doesn't give the particular layout only if i force small size to use inode preallocation i am hitting the problem. ie to change the below line in ext4_mb_group_or_file
if (ac->ac_o_ex.fe_len >= sbi->s_mb_small_req)
to
if (ac->ac_o_ex.fe_len <= sbi->s_mb_small_req)
Do you want to test the patch with this change ?
We are observing the problem with delalloc and nomballoc.
-aneesh

Aneesh Kumar K.V

author

2007-10-29 15:56 UTC #4 lore raw

Quoted 18 lines · Alex Tomas

sorry, I don't quite understand how do you observe this with nomballoc
thanks, Alex
Quoted 13 lines · Aneesh Kumar K.V
mballoc by default doesn't give the particular layout only if i force small size to use inode preallocation i am hitting the problem. ie to change the below line in ext4_mb_group_or_file
if (ac->ac_o_ex.fe_len >= sbi->s_mb_small_req)
to
if (ac->ac_o_ex.fe_len <= sbi->s_mb_small_req)
Do you want to test the patch with this change ?
We are observing the problem with delalloc and nomballoc.

As i explained in the previous mail the problem is with the the current reservation code using ext4_block_alloc_info.

EXT4_I(inode)->i_block_alloc_info;

Now what is happening is we are not discarding the reservation with respect to particular inode in case of dealloc. Without delalloc we discard the reservation during close(). But with dealloc the we are getting new reservation in the writeback path and we don't discard the reservation. This results in the files being spread across and not closely allocated on disk.

BTW with your patch and the change i suggested above the problem still exist.

The output is while requesting for 2 blocks
printed by this in ext4_ext_get_blocks

printk(KERN_CRIT "allocate new block: goal %llu, found %llu/%lu\n",

                      ar.goal, newblock, ar.len);

allocate new block: goal 28672, found 12288/1
allocate new block: goal 8192, found 12292/2
allocate new block: goal 8192, found 12296/2
allocate new block: goal 8192, found 12300/2
allocate new block: goal 8192, found 12304/2
allocate new block: goal 8192, found 12308/2
allocate new block: goal 8192, found 12312/2
allocate new block: goal 8192, found 12316/2
allocate new block: goal 8192, found 1440/2
allocate new block: goal 8192, found 1444/2
allocate new block: goal 8192, found 1448/2
allocate new block: goal 8192, found 1452/2
allocate new block: goal 8192, found 1456/2
allocate new block: goal 8192, found 1460/2
allocate new block: goal 8192, found 1464/2
allocate new block: goal 8192, found 1468/2
allocate new block: goal 8192, found 12320/2
allocate new block: goal 8192, found 12324/2
allocate new block: goal 8192, found 12328/2
allocate new block: goal 8192, found 12332/2
allocate new block: goal 8192, found 12336/2

with the change mballoc was not giving the problem described because it uses blocks from group preallocation.

-aneesh

Alex Tomas

2007-10-29 15:27 UTC #5 lore raw

ah, got it now. I think the solution would be to discard preallocated blocks once blocks for all dirty data are allocated and file is closed. In the previous version of delalloc I did this passing NOPREALLOC hint. something similar should be done in the newer one, I guess.

thanks, Alex

Quoted 73 lines · Aneesh Kumar K.V

Quoted 18 lines · Alex Tomas
sorry, I don't quite understand how do you observe this with nomballoc
thanks, Alex
Quoted 13 lines · Aneesh Kumar K.V
mballoc by default doesn't give the particular layout only if i force small size to use inode preallocation i am hitting the problem. ie to change the below line in ext4_mb_group_or_file
if (ac->ac_o_ex.fe_len >= sbi->s_mb_small_req)
to
if (ac->ac_o_ex.fe_len <= sbi->s_mb_small_req)
Do you want to test the patch with this change ?
We are observing the problem with delalloc and nomballoc.
As i explained in the previous mail the problem is with the the current reservation code using ext4_block_alloc_info.
EXT4_I(inode)->i_block_alloc_info;
Now what is happening is we are not discarding the reservation with respect to particular inode in case of dealloc. Without delalloc we discard the reservation during close(). But with dealloc the we are getting new reservation in the writeback
path and we don't discard the reservation. This results
in the files being spread across and not closely allocated
on disk.
BTW with your patch and the change i suggested above the problem still exist.
The output is while requesting for 2 blocks
printed by this in ext4_ext_get_blocks
printk(KERN_CRIT "allocate new block: goal %llu, found %llu/%lu\n",
                      ar.goal, newblock, ar.len);
allocate new block: goal 28672, found 12288/1
allocate new block: goal 8192, found 12292/2
allocate new block: goal 8192, found 12296/2
allocate new block: goal 8192, found 12300/2
allocate new block: goal 8192, found 12304/2
allocate new block: goal 8192, found 12308/2
allocate new block: goal 8192, found 12312/2
allocate new block: goal 8192, found 12316/2
allocate new block: goal 8192, found 1440/2
allocate new block: goal 8192, found 1444/2
allocate new block: goal 8192, found 1448/2
allocate new block: goal 8192, found 1452/2
allocate new block: goal 8192, found 1456/2
allocate new block: goal 8192, found 1460/2
allocate new block: goal 8192, found 1464/2
allocate new block: goal 8192, found 1468/2
allocate new block: goal 8192, found 12320/2
allocate new block: goal 8192, found 12324/2
allocate new block: goal 8192, found 12328/2
allocate new block: goal 8192, found 12332/2
allocate new block: goal 8192, found 12336/2
with the change mballoc was not giving the problem described because it uses blocks from group preallocation.
-aneesh

Aneesh Kumar K.V

author

2007-10-29 17:03 UTC #6 lore raw

In reply to Aneesh Kumar K.V

Quoted 23 lines · Aneesh Kumar K.V

Hi All,
I looked at the delalloc and reservation differences that Valerie was observing.
Below is my understanding. I am not sure whether the below will result in higher fragmentation that Eric Sandeen is observing. I guess it should not. Even though the reservation gets discarded during the clear inode due to memory pressure
the request for new reservation should get the blocks nearby and not break extents right ?
any how below is the simple case.
without delalloc the blocks are requested during prepare_write/write_begin. That means we enter ext4_new_blocks_old which will call ext4_try_to_allocate_with_rsv. Now if there is no reservation for this inode a new one will be allocated. After using the blocks this reservation is destroyed during the close via ext4_release_file
With delalloc the blocks are not requested until we hit writeback/ext4_da_writepages
That means if we create new file and close them the reservation will be discarded
during close via ext4_release_file.( Actually there will be nothing to clear)
Now when we do a sync/or write back. We try to get the block, the inode will
request for new reservation. This reservation is not discarded untill we call clear_inode
and that results in the behavior we are seeing.
Free blocks: 1440-8191, 8194-8199, 8202-8207, 8210-8215, 8218-8223, 8226-8231, 8234-8239, 8242-8247, 8250-8255, 8258-8263, 8266-8271, 8274-8279, 8282-8287, 8290-8295, 8298-8303, 8306-8311, 8314-8319, 8322-8327, 8330-8335, 8338-8343, 8346-12799

Quoted 3 lines

So now the question is where do we discard the reservation in case of delalloc.
-

with respect to mballoc we are not seeing this because we are doing allocation from group prealloc list which is per cpu.

For most the case we have EXT4_MB_HINT_GROUP_ALLOC set in mballoc.

In ext4_mb_group_or_file i already have a FIXME!! regarding this.

currently we have

       /* request is so large that we don't care about
        * streaming - it overweights any possible seek */
       if (ac->ac_o_ex.fe_len >= sbi->s_mb_large_req)
               return;

       /* FIXME!!
        * is this  >=  considering the above ?
        */
       if (ac->ac_o_ex.fe_len >= sbi->s_mb_small_req)
               return;

       .....
       ......

      /* we're going to use group allocation */
       ac->ac_flags |= EXT4_MB_HINT_GROUP_ALLOC;
       
      ........
      .........

So for small size we have the EXT4_MB_HINT_GROUP_ALLOC set . Now if i change the the line below FIXME!! to <= , that will force small size to use inode prealloc and that cause

Free blocks: 1442-1443, 1446-1447, 1450-1451, 1454-1455, 1458-1459, 1462-1463, 1466-1467, 1470-1471, 1474-1475, 1478-1479, 1482-1483, 1486-1487, 1490-1491, 1494-1495, 1498-1499, 1502-1503, 1506-1507, 1510-1511, 1514-1515, 1518-12799

So the problem is generic.

-aneesh

Aneesh Kumar K.V

author

2007-10-30 11:02 UTC #7 lore raw

[RFC/PATCH] ext4: Clear the reservation window correctly with delayed allocation.

Quoted 57 lines · Aneesh Kumar K.V

Quoted 23 lines · Aneesh Kumar K.V
Hi All,
I looked at the delalloc and reservation differences that Valerie was observing.
Below is my understanding. I am not sure whether the below will result in higher fragmentation that Eric Sandeen is observing. I guess it should not. Even though the reservation gets discarded during the clear inode due to memory pressure
the request for new reservation should get the blocks nearby and not break extents right ?
any how below is the simple case.
without delalloc the blocks are requested during prepare_write/write_begin. That means we enter ext4_new_blocks_old which will call ext4_try_to_allocate_with_rsv. Now if there is no reservation for this inode a new one will be allocated. After using the blocks this reservation is destroyed during the close via ext4_release_file
With delalloc the blocks are not requested until we hit writeback/ext4_da_writepages
That means if we create new file and close them the reservation will be discarded
during close via ext4_release_file.( Actually there will be nothing to clear)
Now when we do a sync/or write back. We try to get the block, the inode will
request for new reservation. This reservation is not discarded untill we call clear_inode
and that results in the behavior we are seeing.
Free blocks: 1440-8191, 8194-8199, 8202-8207, 8210-8215, 8218-8223, 8226-8231, 8234-8239, 8242-8247, 8250-8255, 8258-8263, 8266-8271, 8274-8279, 8282-8287, 8290-8295, 8298-8303, 8306-8311, 8314-8319, 8322-8327, 8330-8335, 8338-8343, 8346-12799
Quoted 3 lines
So now the question is where do we discard the reservation in case of delalloc.
-
with respect to mballoc we are not seeing this because we are doing allocation from group prealloc list which is per cpu. For most the case we have EXT4_MB_HINT_GROUP_ALLOC set in mballoc.
In ext4_mb_group_or_file i already have a FIXME!! regarding this.
currently we have
       /* request is so large that we don't care about
        * streaming - it overweights any possible seek */
       if (ac->ac_o_ex.fe_len >= sbi->s_mb_large_req)
               return;

       /* FIXME!!
        * is this  >=  considering the above ?
        */
       if (ac->ac_o_ex.fe_len >= sbi->s_mb_small_req)
               return;

       .....
       ......

      /* we're going to use group allocation */
       ac->ac_flags |= EXT4_MB_HINT_GROUP_ALLOC;
             ........
      .........

Quoted 9 lines

So for small size we have the EXT4_MB_HINT_GROUP_ALLOC set . Now if i change the the line below FIXME!! to <= , that will force small size to use inode prealloc and that cause
Free blocks: 1442-1443, 1446-1447, 1450-1451, 1454-1455, 1458-1459, 1462-1463, 1466-1467, 1470-1471, 1474-1475, 1478-1479, 1482-1483, 1486-1487, 1490-1491, 1494-1495, 1498-1499, 1502-1503, 1506-1507, 1510-1511, 1514-1515, 1518-12799
So the problem is generic.

the below patch give ok results with nomballoc.

allocate new block: goal 8192, found 8192/2 allocate new block: goal 8192, found 8194/2 allocate new block: goal 8192, found 8196/2 allocate new block: goal 8192, found 8198/2 allocate new block: goal 8192, found 8200/2 allocate new block: goal 8192, found 8202/2 allocate new block: goal 8192, found 8204/2 allocate new block: goal 8192, found 8206/2 allocate new block: goal 8192, found 8208/2 allocate new block: goal 8192, found 8210/2 allocate new block: goal 8192, found 8212/2 allocate new block: goal 8192, found 8214/2 allocate new block: goal 8192, found 8216/2 allocate new block: goal 8192, found 8218/2 allocate new block: goal 8192, found 8220/2 allocate new block: goal 8192, found 8222/2 allocate new block: goal 8192, found 8224/2 allocate new block: goal 8192, found 8226/2 allocate new block: goal 8192, found 8228/2 allocate new block: goal 8192, found 8230/2

fs/ext4/inode.c +0 -0

index ac4d032..a3a7205 100644
@@ -1410,7 +1410,14 @@ out:

static int ext4_da_writepages(struct address_space *mapping,
struct writeback_control *wbc)
{
- return mpage_da_writepages(mapping, wbc, ext4_da_get_block_write);
+ int retval;
+ retval = mpage_da_writepages(mapping, wbc, ext4_da_get_block_write);
+ if (!retval) {
+ /* if writepages is successfull discard the reservation */
+ ext4_discard_reservation(mapping->host);
+ }
+
+ return retval;
}

static void ext4_da_invalidatepage(struct page *page, unsigned long offset)

-aneesh

Alex Tomas

2007-10-30 11:24 UTC #8 lore raw

Re: [RFC/PATCH] ext4: Clear the reservation window correctly with delayed allocation.

what if writepages went through 100 pages only and another 100 are left because negative wbc->nr_to_write?

thanks, Alex

Quoted 24 lines · Aneesh Kumar K.V

(quoted diff) +8 -3

@@ -1410,7 +1410,14 @@ out:
 static int ext4_da_writepages(struct address_space *mapping,
                  struct writeback_control *wbc)
 {
-    return mpage_da_writepages(mapping, wbc, ext4_da_get_block_write);
+    int retval;
+    retval = mpage_da_writepages(mapping, wbc, ext4_da_get_block_write);
+    if (!retval) {
+        /* if writepages is successfull discard the reservation */
+        ext4_discard_reservation(mapping->host);
+    }
+
+    return retval;
 }
 
 static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
 
 
-aneesh
-

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html