delayed allocation for ext4
Subsystems (2)
Good day,
probably the previous set of patches (including mballoc/lg) is too large. so, I reworked delayed allocation a bit so that it can be used on top of regular balloc, though it still can be used with extents-enabled files only.
this time series contains just 3 patches:
- booked-page-flag.patch
adds PG_booked bit to page->flags. it's used in delayed allocation to mark space is already reserved for page (including possible metadata)
- ext4-block-reservation.patch
this is scalable free space management. every time we delay allocation of some page, a space (including metadata) should be reserved
- ext4-delayed-allocation.patch
delayed allocation itself, enabled by "delalloc" mount option. extents support is also required. currently it works only with blocksize=pagesize.
all the patches can be used in ftp://ftp.clusterfs.com/pub/people/alex/2.6.20-rc1/
the series passed basic tests like dd/dbench/fsx.
any comments/questions are very welcome.
thanks, Alex
RFC 1/1 booked-page-flag.patch
include/linux/page-flags.h +5 -0
@@ -90,6 +90,7 @@#define PG_reclaim 17/* To be reclaimed asap */ #define PG_nosave_free 18/* Used for system suspend/resume */ #define PG_buddy 19/* Page is free, on buddy lists */ +#define PG_booked 20/* Has blocks reserved on-disk */ #if (BITS_PER_LONG >32 )
@@ -230,6 +231,10 @@ static inline void SetPageUptodate(struc#define SetPageMappedToDisk (page) set_bit(PG_mappedtodisk, &(page)->flags)#define ClearPageMappedToDisk (page) clear_bit(PG_mappedtodisk, &(page)->flags) +#define PageBooked (page) test_bit(PG_booked, &(page)->flags) +#define SetPageBooked (page) set_bit(PG_booked, &(page)->flags) +#define ClearPageBooked (page) clear_bit(PG_booked, &(page)->flags) +#define PageReclaim (page) test_bit(PG_reclaim, &(page)->flags)#define SetPageReclaim (page) set_bit(PG_reclaim, &(page)->flags)#define ClearPageReclaim (page) clear_bit(PG_reclaim, &(page)->flags)
RFC 1/1 ext4-block-reservation.patch
In reply to Alex Tomasinclude/linux/ext4_fs.h +5 -0
@@ -201,6 +201,7 @@ struct ext4_group_desc#define EXT4_STATE_JDATA 0x00000001/* journaled data exists */ #define EXT4_STATE_NEW 0x00000002/* inode is newly created */ #define EXT4_STATE_XATTR 0x00000004/* has in-inode xattrs */ +#define EXT4_STATE_BLOCKS_RESERVED 0x00000008/* blocks reserved */ /* Used to pass group descriptor data when online resize is done */ struct ext4_new_group_input {
@@ -808,6 +809,10 @@ extern struct ext4_group_desc * ext4_getextern int ext4_should_retry_alloc (struct super_block *sb,int *retries);extern void ext4_init_block_alloc_info (struct inode *);extern void ext4_rsv_window_add (struct super_block *sb,struct ext4_reserve_window_node *rsv); +int ext4_reserve_init (struct super_block *sb); +void ext4_reserve_release (struct super_block *sb); +void ext4_release_blocks (struct super_block *sb,int blocks); +int ext4_reserve_blocks (struct super_block *sb,int blocks);/* dir.c */ extern int ext4_check_dir_entry(const char *, struct inode *,
include/linux/ext4_fs_sb.h +5 -0
@@ -24,6 +24,8 @@#endif #include <linux/rbtree.h> +struct ext4_reservation_slot ; +/* * third extended-fs super-block data in memory */
@@ -65,6 +67,9 @@ struct ext4_sb_info {struct rb_root s_rsv_window_root;struct ext4_reserve_window_node s_rsv_window_head; +/* global reservation structures */ +struct ext4_reservation_slot *s_reservation_slots; +/* Journaling */ struct inode * s_journal_inode;struct journal_s * s_journal;
fs/ext4/super.c +2 -0
@@ -439,6 +439,7 @@ static void ext4_put_super (struct superstruct ext4_super_block *es = sbi->s_es;int i; +ext4_reserve_release (sb);ext4_ext_release (sb);ext4_xattr_put_super (sb);jbd2_journal_destroy (sbi->s_journal);
@@ -1867,6 +1868,7 @@ static int ext4_fill_super (struct super "writeback");ext4_ext_init (sb); +ext4_reserve_init (sb);lock_kernel ();return 0 ;
fs/ext4/balloc.c +173 -3
205 lines not on this page · raw message
Re: [RFC] ext4-block-reservation.patch
On Fri, 22 Dec 2006 23:25:16 +0300
Alex Tomas <[email protected]> wrote:
Once this code is settled in we should consider removal of the existing reservations code from ext4.
fs/ext4/balloc.c unsigned long ext4_bg_num_gdb(struct sup · line 1868
Quoted 5 lines
fs/ext4/balloc.c +5
-0+ +struct ext4_reservation_slot { +__u64 rs_reserved; +spinlock_t rs_lock; +}____cacheline_aligned ;
Should be ____cacheline_aligned_in_smp.
That's assuming it needs to be cacheline aligned at all. It can consume a lot of space.
<looks>
oh, this should be allocated with alloc_percpu(), in which case the open-coded alignment can perhaps go away.
fs/ext4/balloc.c unsigned long ext4_bg_num_gdb(struct sup · line 1878
Quoted 9 lines
fs/ext4/balloc.c +9
-0+ +int ext4_reserve_local (struct super_block *sb,int blocks) +{ +struct ext4_sb_info *sbi =EXT4_SB (sb); +struct ext4_reservation_slot *rs; +int rc = -ENOSPC ; + +preempt_disable (); + rs = sbi->s_reservation_slots +smp_processor_id ();
use get_cpu() here.
Quoted 6 lines
fs/ext4/balloc.c +6
-0+void ext4_rebalance_reservation (struct ext4_reservation_slot *rs,__u64 free) +{ +int i, used_slots =0 ; +__u64 chunk; + +/* let's know what slots have been used */
fs/ext4/balloc.c unsigned long ext4_bg_num_gdb(struct sup · line 1907
Quoted 10 lines
fs/ext4/balloc.c +10
-0+for (i =0 ; i <NR_CPUS ; i++) +if (rs[i].rs_reserved || i ==smp_processor_id ()) + used_slots++; + +/* chunk is a number of block every used +* slot will get. make sure it isn't 0 */ + chunk = free + used_slots -1 ; +do_div (chunk, used_slots); + +for (i =0 ; i <NR_CPUS ; i++) {
all these NR_CPUS loops need to go away. Use either for_each_possible_cpu() or, preferably, for_each_online_cpu() and a hotplug notifier.
Why is this code using per-cpu data at all, btw? These optimisations tend to be marginal in filesystems. What is the perfomance impact of making this data be single-superblock-wide-instance?
fs/ext4/balloc.c unsigned long ext4_bg_num_gdb(struct sup · line 1982
Quoted 7 lines
fs/ext4/balloc.c +7
-0+int ext4_reserve_init (struct super_block *sb) +{ +struct ext4_sb_info *sbi =EXT4_SB (sb); +struct ext4_reservation_slot *rs; +int i; + + rs =kmalloc (sizeof (struct ext4_reservation_slot ) *NR_CPUS ,GFP_KERNEL );
alloc_percpu()
Re: [RFC] ext4-block-reservation.patch
Hi,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Andrew Morton (AM) writes:
AM> Should be ____cacheline_aligned_in_smp.
AM> That's assuming it needs to be cacheline aligned at all. It can consume a AM> lot of space.
the idea is to make block reservation cheap because it's called for every page.
AM> <looks>
AM> oh, this should be allocated with alloc_percpu(), in which case the AM> open-coded alignment can perhaps go away.
got it.
Quoted 9 lines
fs/ext4/balloc.c unsigned long ext4_bg_num_gdb(struct sup · line 1878
Quoted 9 lines
fs/ext4/balloc.c +9
-0+ +int ext4_reserve_local (struct super_block *sb,int blocks) +{ +struct ext4_sb_info *sbi =EXT4_SB (sb); +struct ext4_reservation_slot *rs; +int rc = -ENOSPC ; + +preempt_disable (); + rs = sbi->s_reservation_slots +smp_processor_id ();
AM> use get_cpu() here.
ok.
Quoted 8 lines
Quoted 8 lines
fs/ext4/balloc.c +8
-0+void ext4_rebalance_reservation (struct ext4_reservation_slot *rs,__u64 free) +{ +int i, used_slots =0 ; +__u64 chunk; + +/* let's know what slots have been used */ +for (i =0 ; i <NR_CPUS ; i++) +if (rs[i].rs_reserved || i ==smp_processor_id ())
Quoted 8 lines
fs/ext4/balloc.c unsigned long ext4_bg_num_gdb(struct sup · line 1907
Quoted 8 lines
fs/ext4/balloc.c +8
-0+ used_slots++; + +/* chunk is a number of block every used +* slot will get. make sure it isn't 0 */ + chunk = free + used_slots -1 ; +do_div (chunk, used_slots); + +for (i =0 ; i <NR_CPUS ; i++) {
AM> all these NR_CPUS loops need to go away. Use either
AM> for_each_possible_cpu() or, preferably, for_each_online_cpu() and a hotplug AM> notifier.
hmm, i see.
AM> Why is this code using per-cpu data at all, btw? These optimisations tend AM> to be marginal in filesystems. What is the perfomance impact of making AM> this data be single-superblock-wide-instance?
well, even on 2way box a single-lock reservation was in top10.
thanks, Alex
RFC 1/1 ext4-delayed-allocation.patch
In reply to Alex Tomasinclude/linux/ext4_fs_i.h +5 -0
@@ -153,6 +153,11 @@ struct ext4_inode_info {unsigned long i_ext_generation;struct ext4_ext_cache i_cached_extent; + +__u32 i_blocks_reserved; +__u32 i_md_reserved; +spinlock_t i_wb_reserved_lock;/* to protect i_md_reserved */ + atomic_t i_wb_writers; };#endif /* _LINUX_EXT4_FS_I */
include/linux/ext4_fs.h +13 -0
@@ -401,6 +401,7 @@ struct ext4_inode {#define EXT4_MOUNT_USRQUOTA 0x100000/* "old" user quota */ #define EXT4_MOUNT_GRPQUOTA 0x200000/* "old" group quota */ #define EXT4_MOUNT_EXTENTS 0x400000/* Extents support */ +#define EXT4_MOUNT_DELAYED_ALLOC 0x1000000/* Delayed allocation support */ /* Compatibility, for having both ext2_fs.h and ext4_fs.h included at once */ #ifndef _LINUX_EXT2_FS_H
@@ -994,6 +995,18 @@ ext4_get_blocks_wrap(handle_t *handle, s } +/* writeback.c */ +extern int ext4_wb_writepages (struct address_space *,struct writeback_control *); +extern int ext4_wb_prepare_write (struct file *file,struct page *page, +unsigned from,unsigned to); +extern int ext4_wb_commit_write (struct file *,struct page *,unsigned ,unsigned ); +extern int ext4_wb_writepage (struct page *,struct writeback_control *); +extern void ext4_wb_invalidatepage (struct page *,unsigned long ); +extern int ext4_wb_releasepage (struct page *,gfp_t ); +extern int ext4_wb_block_truncate_page (handle_t *,struct page *,struct address_space *,loff_t ); +extern void ext4_wb_init (struct super_block *); +extern void ext4_wb_release (struct super_block *); +#endif /* __KERNEL__ */ #endif /* _LINUX_EXT4_FS_H */
include/linux/ext4_fs_sb.h +11 -0
@@ -94,6 +94,17 @@ struct ext4_sb_info {unsigned long s_ext_blocks;unsigned long s_ext_extents;#endif + +atomic_t s_wb_congested; +atomic_t s_wb_single_pages; +atomic_t s_wb_collisions_sp; +atomic_t s_wb_allocated; +atomic_t s_wb_reqs; +atomic_t s_wb_nr_to_write; +atomic_t s_wb_collisions; +atomic_t s_wb_blocks; +atomic_t s_wb_extents; + atomic_t s_wb_dropped; };#endif /* _LINUX_EXT4_FS_SB */
include/linux/ext4_fs_extents.h +1 -0
@@ -193,6 +193,7 @@ extern int ext4_ext_calc_credits_for_insextern int ext4_ext_insert_extent (handle_t *,struct inode *,struct ext4_ext_path *,struct ext4_extent *);extern int ext4_ext_walk_space (struct inode *,unsigned long ,unsigned long ,ext_prepare_callback ,void *);extern struct ext4_ext_path *ext4_ext_find_extent (struct inode *,int ,struct ext4_ext_path *); +int ext4_ext_calc_metadata_amount (struct inode *inode,int blocks);#endif /* _LINUX_EXT4_EXTENTS */
fs/ext4/super.c +14 -1
@@ -439,6 +439,7 @@ static void ext4_put_super (struct superstruct ext4_super_block *es = sbi->s_es;int i; +ext4_wb_release (sb);ext4_reserve_release (sb);ext4_ext_release (sb);ext4_xattr_put_super (sb);
@@ -506,6 +507,13 @@ static struct inode *ext4_alloc_inode(st ei->i_block_alloc_info =NULL ; ei->vfs_inode.i_version =1 ;memset (&ei->i_cached_extent,0 ,sizeof (struct ext4_ext_cache )); + +/* FIXME: these wb-related fields could be initialized once */ + ei->i_blocks_reserved =0 ; + ei->i_md_reserved =0 ; +atomic_set (&ei->i_wb_writers,0 ); +spin_lock_init (&ei->i_wb_reserved_lock); +return &ei->vfs_inode; }
@@ -729,7 +737,7 @@ enum { Opt_usrjquota, Opt_grpjquota, Opt_offusrjquota, Opt_offgrpjquota, Opt_jqfmt_vfsold, Opt_jqfmt_vfsv0, Opt_quota, Opt_noquota, Opt_ignore, Opt_barrier, Opt_err, Opt_resize, Opt_usrquota, - Opt_grpquota, Opt_extents, + Opt_grpquota, Opt_extents, Opt_delayed_alloc, }; static match_table_t tokens = {
@@ -780,6 +788,7 @@ static match_table_t tokens = { {Opt_usrquota, "usrquota"}, {Opt_barrier, "barrier=%u"}, {Opt_extents, "extents"}, + {Opt_delayed_alloc, "delalloc"}, {Opt_err, NULL}, {Opt_resize, "resize"}, };
@@ -1094,6 +1103,9 @@ clear_qf_name:else clear_opt (sbi->s_mount_opt,BARRIER );break ; +case Opt_delayed_alloc: +set_opt (sbi->s_mount_opt,DELAYED_ALLOC ); +break ;case Opt_ignore:break ;case Opt_resize:
@@ -1869,6 +1881,7 @@ static int ext4_fill_super (struct superext4_ext_init (sb);ext4_reserve_init (sb); +ext4_wb_init (sb);lock_kernel ();return 0 ;
fs/ext4/extents.c +30 -0
@@ -2159,6 +2159,36 @@ int ext4_ext_writepage_trans_blocks(strureturn needed; } +int ext4_ext_calc_metadata_amount (struct inode *inode,int blocks) +{ +int lcap, icap, rcap, leafs, idxs, num; + + rcap =ext4_ext_space_root (inode); +if (blocks <= rcap) { +/* all extents fit to the root */ +return 0 ; + } + + rcap =ext4_ext_space_root_idx (inode); + lcap =ext4_ext_space_block (inode); + icap =ext4_ext_space_block_idx (inode); + + num = leafs = (blocks + lcap -1 ) / lcap; +if (leafs <= rcap) { +/* all pointers to leafs fit to the root */ +return leafs; + } + +/* ok. we need separate index block(s) to link all leaf blocks */ + idxs = (leafs + icap -1 ) / icap; +do { + num += idxs; + idxs = (idxs + icap -1 ) / icap; + }while (idxs > rcap); + +return num; +} +EXPORT_SYMBOL (ext4_mark_inode_dirty);EXPORT_SYMBOL (ext4_ext_invalidate_cache);EXPORT_SYMBOL (ext4_ext_insert_extent);
fs/ext4/Makefile +1 -1
@@ -6,7 +6,7 @@ obj-$(CONFIG_EXT4DEV_FS) += ext4dev.o ext4dev-y := balloc.o bitmap.o dir.o file.o fsync.o ialloc.o inode.o \ ioctl.o namei.o super.o symlink.o hash.o resize.o extents.o \ - ext4_jbd2.o + ext4_jbd2.o writeback.o ext4dev-$(CONFIG_EXT4DEV_FS_XATTR) += xattr.o xattr_user.o xattr_trusted.o ext4dev-$(CONFIG_EXT4DEV_FS_POSIX_ACL) += acl.o
fs/ext4/writeback.c +1167 -0
1,167 lines not on this page · raw message
fs/ext4/file.c +2 -2
@@ -35,8 +35,8 @@ static int ext4_release_file (struct ino {/* if we are the last writer on the inode, drop the block reservation */ if ((filp->f_mode &FMODE_WRITE ) && - (atomic_read (&inode->i_writecount) ==1 )) - { + (atomic_read (&inode->i_writecount) ==1 ) && +EXT4_I (inode)->i_blocks_reserved ==0 ) {mutex_lock (&EXT4_I (inode)->truncate_mutex);ext4_discard_reservation (inode);mutex_unlock (&EXT4_I (inode)->truncate_mutex);
fs/ext4/inode.c +32 -2
@@ -943,7 +943,7 @@ out:#define DIO_CREDITS (EXT4_RESERVE_TRANS_BLOCKS + 32) -static int ext4_get_block (struct inode *inode,sector_t iblock, +int ext4_get_block (struct inode *inode,sector_t iblock,struct buffer_head *bh_result,int create) {handle_t *handle =journal_current_handle ();
@@ -1807,9 +1807,34 @@ static const struct address_space_operat .releasepage = ext4_releasepage, }; +static int ext4_wb_set_page_dirty (struct page *page) +{ +return __set_page_dirty_nobuffers (page); +} + +static struct address_space_operations ext4_writeback_da_aops = { + .readpage = ext4_readpage, + .readpages = ext4_readpages, + .writepage = ext4_wb_writepage, + .writepages = ext4_wb_writepages, + .sync_page = block_sync_page, + .prepare_write = ext4_wb_prepare_write, + .commit_write = ext4_wb_commit_write, + .bmap = ext4_bmap, + .invalidatepage = ext4_wb_invalidatepage, + .releasepage = ext4_wb_releasepage, + .set_page_dirty = ext4_wb_set_page_dirty, + .direct_IO = ext4_direct_IO, +}; +void ext4_set_aops (struct inode *inode) { -if (ext4_should_order_data (inode)) +if (S_ISREG (inode->i_mode) && + (EXT4_I (inode)->i_flags &EXT4_EXTENTS_FL ) && +test_opt (inode->i_sb,EXTENTS ) && +test_opt (inode->i_sb,DELAYED_ALLOC )) + inode->i_mapping->a_ops = &ext4_writeback_da_aops; +else if (ext4_should_order_data (inode)) inode->i_mapping->a_ops = &ext4_ordered_aops;else if (ext4_should_writeback_data (inode)) inode->i_mapping->a_ops = &ext4_writeback_aops;
@@ -1834,6 +1859,11 @@ int ext4_block_truncate_page(handle_t *hint err =0 ;void *kaddr; +if ((EXT4_I (inode)->i_flags &EXT4_EXTENTS_FL ) && +test_opt (inode->i_sb,EXTENTS ) && +test_opt (inode->i_sb,DELAYED_ALLOC )) +return ext4_wb_block_truncate_page (handle, page, mapping, from); + blocksize = inode->i_sb->s_blocksize; length = blocksize - (offset & (blocksize -1 )); iblock = index << (PAGE_CACHE_SHIFT - inode->i_sb->s_blocksize_bits);
Re: [RFC] ext4-delayed-allocation.patch
fs/ext4/writeback.c line 69
Quoted 8 lines · On Fri, 22 Dec 2006 23:28:32 +0300 Alex Tomas <[email protected]> wrote:
fs/ext4/writeback.c +8
-0+/* +* With EXT4_WB_SKIP_SMALL defined the patch will try to avoid +* small I/Os ignoring ->writepages() if mapping hasn't enough +* contig. dirty pages +*/ +#define EXT4_WB_SKIP_SMALL__ + +#define WB_ASSERT (__x__) if (!(__x__)) BUG();
This is unused. Please kill.
fs/ext4/writeback.c line 76
Quoted 6 lines
fs/ext4/writeback.c +6
-0+#define WB_DEBUG__ +#ifdef WB_DEBUG +#define wb_debug(fmt,a...) printk(fmt, ##a); +#else +#define wb_debug(fmt,a...) +#endif
It's tiresome for each piece of kernel code to implement private debug macros. Why not use pr_debug()?
In general, this patch adds a mountain of code which I suspect just shouldn't be in a filesystem. It should be library code in mm/ or fs/. It'll take some thought and refactoring and definition of new address_space_operations entries, etc. But burying all this inside a single filesystem is just The Wrong Thing To Do.
I am not inclined to review the code in detail because the lack of suitable code comments would make that task much larger and significantly less useful than it should be. Please spend a day or so commenting the code in a similar manner to other parts of ext4 and jbd2. When doing so, put yourself in the position of an experienced kernel developer who seeks to understand what the code is doing and why it is doing it. Skilful commenting is essential if the code is to be maintainable and it has an immediate impact upon the code's quality. Every non-trivial function should have an introductory comment which tells the reader why this function exists, what it does and, if not obvious, how it does it. Don't bother with kernel-doc comments - for some reason they tend to be useless for code conprehension.
I shouldn't need to sit here scratching my head and wondering why the heck a writepages function is running __set_page_dirty_nobuffers().
Thanks.
Re: [RFC] ext4-delayed-allocation.patch
In reply to Alex TomasHi Alex,
I tested your patch on my system with a 20TB device, but some tests failed. Looking at the code, I saw that the support of 48-bit block number in extents is lacking.
I made some changes in the code (see the patch in attachment) and now all my tests are OK. The patch is not complete, I didn't update calls to wb_debug() which dump "ee_start".
Hope this helps.
Valérie
fs/ext4/writeback.c +13 -6
@@ -274,7 +274,8 @@ static int ext4_wb_submit_extent(structstruct inode *inode = wc->mapping->host;int blkbits = inode->i_blkbits;struct page *page; -unsigned long blk, off, len, remain; +ext4_fsblk_t off; +unsigned long blk, len, remain;unsigned long pstart, plen, prev;struct bio *bio =NULL ;int nr_pages;
@@ -332,6 +333,7 @@ alloc_new_bio: nr_pages = (ex->ee_len - (blk - ex->ee_block)); nr_pages = (nr_pages >> (PAGE_CACHE_SHIFT - blkbits)); off = ex->ee_start + (blk - ex->ee_block); + off |= (ext4_fsblk_t ) ex->ee_start_hi <<32 ; bio =ext4_wb_bio_alloc (inode, off, nr_pages +2 );if (bio ==NULL )return -ENOMEM ;
@@ -377,7 +379,9 @@ ext4_wb_find_goal(struct inode *inode, s/* try to predict block placement */ if ((ex = path[depth].p_ext)) -return ex->ee_start + (block - ex->ee_block); +return ((ex->ee_start + | ((ext4_fsblk_t ) ex->ee_start_hi <<32 )) + + (block - ex->ee_block);/* it looks index is empty * try to find starting from index itself */
@@ -416,7 +420,8 @@ static int ext4_wb_handle_extent(struct (unsigned ) ec->ec_block, (unsigned ) ec->ec_len, (unsigned) ec->ec_start); - nex.ee_start = ec->ec_start; + nex.ee_start = ec->ec_start &0xffffffff ; + nex.ee_start_hi = (ec->ec_start >>32 ) &0xffff ; nex.ee_block = ec->ec_block; nex.ee_len = ec->ec_len; err =ext4_wb_submit_extent (wc,NULL , &nex,0 );
@@ -488,8 +493,8 @@ static int ext4_wb_handle_extent(struct pblock, count, inode->i_ino, ec->ec_len);/* insert new extent */ - nex.ee_start = pblock; - nex.ee_start_hi =0 ; + nex.ee_start = pblock &0xffffffff ; + nex.ee_start_hi = (pblock >>32 ) &0xffff ; nex.ee_len = count; nex.ee_block = ec->ec_block; err =ext4_ext_insert_extent (handle, inode, path, &nex);
@@ -520,7 +525,9 @@ static int ext4_wb_handle_extent(struct/* block have been allocated for data, so time to drop dirty * in correspondend buffer_heads to prevent corruptions */ for (i =0 ; i < nex.ee_len; i++) -unmap_underlying_metadata (sb->s_bdev, nex.ee_start + i); +unmap_underlying_metadata (sb->s_bdev, + ((ext4_fsblk_t ) nex.ee_start_hi <<32 ) + + nex.ee_start + i);/* correct on-disk inode size */ if (nex.ee_len >0 ) {
Re: [RFC] ext4-delayed-allocation.patch
ah, you're right. thanks!
thanks, Alex
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Valerie Clement (VC) writes:
VC> Hi Alex,
VC> I tested your patch on my system with a 20TB device, but some tests VC> failed. Looking at the code, I saw that the support of 48-bit block VC> number in extents is lacking.
VC> I made some changes in the code (see the patch in attachment) and now VC> all my tests are OK.
VC> The patch is not complete, I didn't update calls to wb_debug() which VC> dump "ee_start".
VC> Hope this helps.
VC> Valérie
VC> Index: linux-2.6.20-rc1/fs/ext4/writeback.c
VC> ===================================================================
VC> --- linux-2.6.20-rc1.orig/fs/ext4/writeback.c 2007-01-09 18:41:27.000000000 +0100
VC> +++ linux-2.6.20-rc1/fs/ext4/writeback.c 2007-01-11 13:25:38.000000000 +0100
VC> @@ -274,7 +274,8 @@ static int ext4_wb_submit_extent(struct
VC> struct inode *inode = wc->mapping->host; VC> int blkbits = inode->i_blkbits; VC> struct page *page; VC> - unsigned long blk, off, len, remain; VC> + ext4_fsblk_t off; VC> + unsigned long blk, len, remain; VC> unsigned long pstart, plen, prev; VC> struct bio *bio = NULL; VC> int nr_pages; VC> @@ -332,6 +333,7 @@ alloc_new_bio: VC> nr_pages = (ex->ee_len - (blk - ex->ee_block)); VC> nr_pages = (nr_pages >> (PAGE_CACHE_SHIFT - blkbits)); VC> off = ex->ee_start + (blk - ex->ee_block); VC> + off |= (ext4_fsblk_t) ex->ee_start_hi << 32; VC> bio = ext4_wb_bio_alloc(inode, off, nr_pages + 2); VC> if (bio == NULL) VC> return -ENOMEM;
VC> @@ -377,7 +379,9 @@ ext4_wb_find_goal(struct inode *inode, s
VC> /* try to predict block placement */
VC> if ((ex = path[depth].p_ext))
VC> - return ex->ee_start + (block - ex->ee_block);
VC> + return ((ex->ee_start
VC> + | ((ext4_fsblk_t) ex->ee_start_hi << 32))
VC> + + (block - ex->ee_block);
VC> /* it looks index is empty
VC> * try to find starting from index itself */
VC> @@ -416,7 +420,8 @@ static int ext4_wb_handle_extent(struct
VC> (unsigned) ec->ec_block, VC> (unsigned) ec->ec_len, VC> (unsigned) ec->ec_start); VC> - nex.ee_start = ec->ec_start; VC> + nex.ee_start = ec->ec_start & 0xffffffff; VC> + nex.ee_start_hi = (ec->ec_start >> 32) & 0xffff; VC> nex.ee_block = ec->ec_block; VC> nex.ee_len = ec->ec_len; VC> err = ext4_wb_submit_extent(wc, NULL, &nex, 0);
VC> @@ -488,8 +493,8 @@ static int ext4_wb_handle_extent(struct
VC> pblock, count, inode->i_ino, ec->ec_len);
VC> /* insert new extent */
VC> - nex.ee_start = pblock;
VC> - nex.ee_start_hi = 0;
VC> + nex.ee_start = pblock & 0xffffffff;
VC> + nex.ee_start_hi = (pblock >> 32) & 0xffff;
VC> nex.ee_len = count; VC> nex.ee_block = ec->ec_block; VC> err = ext4_ext_insert_extent(handle, inode, path, &nex); VC> @@ -520,7 +525,9 @@ static int ext4_wb_handle_extent(struct VC> /* block have been allocated for data, so time to drop dirty VC> * in correspondend buffer_heads to prevent corruptions */ VC> for (i = 0; i < nex.ee_len; i++)
VC> - unmap_underlying_metadata(sb->s_bdev, nex.ee_start + i);
VC> + unmap_underlying_metadata(sb->s_bdev,
VC> + ((ext4_fsblk_t) nex.ee_start_hi << 32)
VC> + + nex.ee_start + i);
VC> /* correct on-disk inode size */
VC> if (nex.ee_len > 0) {
Quoted 8 lines · Alex Tomas
Good day,
probably the previous set of patches (including mballoc/lg) is too large. so, I reworked delayed allocation a bit so that it can be used on top of regular balloc, though it still can be used with extents-enabled files only.
Quoted 6 lines
this time series contains just 3 patches:
- booked-page-flag.patch
addsPG_bookedbit topage->flags. it's used in delayed allocation to mark space is already reserved for page (including possible metadata)
So that mean's we'll have 2 separate mechanisms for marking pages as delalloc. XFS uses the BH_delay flag to indicate that a buffer (block) attached to the page is using delalloc.
FWIW, how does this mechanism deal with block size < page size? Don't you have to track delalloc on a block basis rather than a page basis?
Quoted 9 lines
- ext4-block-reservation.patch
this is scalable free space management. every time we delay allocation of some page, a space (including metadata) should be reserved- ext4-delayed-allocation.patch
delayed allocation itself, enabled by "delalloc" mount option. extents support is also required. currently it works only with blocksize=pagesize.
Ah, that's why you can get away with a page flag - you've ignored the partial page delay state problem. Any plans to use the existing method in the future so we will be able to use ext4 delalloc on machines with a page size larger than 4k?
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 9 lines · David Chinner
Quoted 4 lines
- ext4-delayed-allocation.patch
delayed allocation itself, enabled by "delalloc" mount option. extents support is also required. currently it works only with blocksize=pagesize.Ah, that's why you can get away with a page flag - you've ignored the partial page delay state problem. Any plans to use the existing method in the future so we will be able to use ext4 delalloc on machines with a page size larger than 4k?
I think fixing this up for blocksize < pagesize is an absolute requirement to get things merged. We don't need more filesystems that are crippled on half of our platforms.
Note that recording delayed alloc state at a page granularity in addition to just the buffer heads has a lot of advantages aswell and would help xfs, too. But I think it makes a lot more sense to record it as a radix tree tag to speed up the gang lookups for delalloc conversion.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Christoph Hellwig (CH) writes:
CH> Note that recording delayed alloc state at a page granularity in addition CH> to just the buffer heads has a lot of advantages aswell and would help CH> xfs, too. But I think it makes a lot more sense to record it as a radix CH> tree tag to speed up the gang lookups for delalloc conversion.
please, exaplein about radix tree tag. in ext4-delalloc I use this bit the only way - to avoid multiple reservation space for same page. I guess you need to find non-allocated pages. probably to flush them and update number of reserved blocks in case of -ENOSPC?
thanks, Alex
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · David Chinner
Quoted 4 lines
- ext4-delayed-allocation.patch
delayed allocation itself, enabled by "delalloc" mount option. extents support is also required. currently it works only with blocksize=pagesize.Ah, that's why you can get away with a page flag - you've ignored the partial page delay state problem. Any plans to use the existing method in the future so we will be able to use ext4 delalloc on machines with a page size larger than 4k?
Quoted 8 lines
I think fixing this up for blocksize < pagesize is an absolute requirement to get things merged. We don't need more filesystems that are crippled on half of our platforms.
Note that recording delayed alloc state at a page granularity in addition to just the buffer heads has a lot of advantages aswell and would help xfs, too. But I think it makes a lot more sense to record it as a radix tree tag to speed up the gang lookups for delalloc conversion.
I'm not sure it will make that much difference, really. Looking up by delalloc tag is only going to save a few tail pages in pagevec we use for the look up and could be more expensive if delalloc pages are sparsely distributed through the file.
We'd still have to keep the bufferheads around for partial page state, and that becomes an interesting exercise in keeping things coherent between the radix tree and the buffer heads.
Of course, then there's the unwritten state that XFS also carries around per block (bufferhead) which has all the same issues as the delalloc state. I'd hate to have a generic method for handling delalloc state which is different from the handling of the unwritten state and needing two different sets of code to handle what is essentially the same thing....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Good day,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
David Chinner (DC) writes:
DC> So that mean's we'll have 2 separate mechanisms for marking DC> pages as delalloc. XFS uses the BH_delay flag to indicate DC> that a buffer (block) attached to the page is using delalloc.
well, for blocksize=pagesize we can save 56 bytes on every page.
DC> FWIW, how does this mechanism deal with block size < page size? DC> Don't you have to track delalloc on a block basis rather than DC> a page basis?
I'm still thinking how better to deal with that w/o much code duplication.
DC> Ah, that's why you can get away with a page flag - you've ignored DC> the partial page delay state problem. Any plans to use the DC> existing method in the future so we will be able to use ext4 delalloc DC> on machines with a page size larger than 4k?
what do you mean by "exsiting"? BH_delay?
thanks, Alex
Quoted 10 lines · Alex Tomas
Good day,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
David Chinner (DC) writes:
DC> So that mean's we'll have 2 separate mechanisms for marking DC> pages as delalloc. XFS uses the
BH_delayflag to indicate DC> that a buffer (block) attached to the page is using delalloc.well, for blocksize=pagesize we can save 56 bytes on every page.
Sure, but it means that ext4 w/ delalloc won't work on lots of machines....
Quoted 5 lines
DC> FWIW, how does this mechanism deal with block size < page size? DC> Don't you have to track delalloc on a block basis rather than DC> a page basis?
I'm still thinking how better to deal with that w/o much code duplication.
Code duplication in ext4, or across all filesystems?
Quoted 6 lines
DC> Ah, that's why you can get away with a page flag - you've ignored DC> the partial page delay state problem. Any plans to use the DC> existing method in the future so we will be able to use ext4 delalloc DC> on machines with a page size larger than 4k?
what do you mean by "exsiting"?
BH_delay?
Yes.
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
David Chinner (DC) writes:
DC> So that mean's we'll have 2 separate mechanisms for marking DC> pages as delalloc. XFS uses the BH_delay flag to indicate DC> that a buffer (block) attached to the page is using delalloc.
Quoted 2 lines
Quoted 2 lines
well, for blocksize=pagesize we can save 56 bytes on every page.
DC> Sure, but it means that ext4 w/ delalloc won't work on lots of DC> machines....
it does not currenly. but I'm going to implement that. not sure whether it's worth to have two different codepaths for block size=page size and block size < page size.
DC> FWIW, how does this mechanism deal with block size < page size? DC> Don't you have to track delalloc on a block basis rather than DC> a page basis?
Quoted 2 lines
Quoted 2 lines
I'm still thinking how better to deal with that w/o much code duplication.
DC> Code duplication in ext4, or across all filesystems?
given what Andrew said about moving the code into VFS, it's more about all filesystems.
thanks, Alex