Ocfs2 allocation reservations
Subsystems (2)
cover Ocfs2 allocation reservations
Changes from the last patch set:
- added a check for overlapping reservations in ocfs2_resv_insert()
- cleaned up the comments in ocfs2_cannibalize_resv()
- the check for reservation past bitmap end in ocfs2_check_resmap() is more strict now.
- removed the unused m_search_start member of ocfs2_reservation_map
- optimized __ocfs2_resv_find_window() to ignore regions that are too small for the current alloc
- major cleanup of ocfs2_resmap_claimed_bits()
- added a set of BUG_ON's to in ocfs2_resmap_claimed_bits() to check that the passed allocation range is within the window.
- fixed ocfs2_local_alloc_find_clear_bits() to return actual bits allocated
- add a check for a null data_ac in ocfs2_write_begin_nolock()
I also added a fifth patch, "ocfs2: remove ocfs2_local_alloc_in_range()". I could spin this as it's own patch to go upstream earlier if we want.
Finally, thanks to Tao for an excellent review that helped me catch most of those issues.
Original introduction message follows:
The following patches comprise my latest work on enabling larger contiguous allocations in Ocfs2 in the presence of multiple threads. The patches have been through much more testing since my last round. At this point, I'd say they're ready for wider consumption and of course, more review :)
Similarly to the last series, reservations only operate on the local alloc bitmap. The code knows nothing of inodes and allocators however, so we can extend it to the global bitmap (should the need arise) at a future date.
Changes from the last series are numerous. The biggest one however, is that reservations (when enabled) are no longer 'advisory' and represent an actual region of free bits in the local alloc file. The local alloc code obeys reservations unconditionally.
The reason I made this change is because I saw a breakdown in allocation (back to worst-case) on longer running tests, or those with many threads. Those tests it turned out, were exposing "corner cases" in the code where reservations could no longer be honored due to bits having been set in the local alloc bitmap. Better window replacement (and tracking) policy became quite convoluted when the state of the local alloc bitmap wasn't quite known. It is far simpler to just consult the bitmap for windows, and my testing results showed that it worked better too.
This differs from file systems like ext4 (which I used for inspiration), but our allocation strategy differs greatly. Whereas ext4 may have many different block groups in play during a multi-threaded write we only have the single (and relatively smaller) local alloc window. Reservations can afford to be advisory for ext4, in Ocfs2 however we need them to be honored.
As for results, I provide one of my recent test runs on a 4k/4k file system:
dd if=/dev/urandom of=/ocfs2/1 bs=4096 count=10000 & dd if=/dev/urandom of=/ocfs2/2 bs=4096 count=10000 & dd if=/dev/urandom of=/ocfs2/3 bs=4096 count=10000 &
resv_level=0
Inode: 16920 % fragmented: 93.48 clusters: 10000 extents: 9348 score: 23931
Inode: 16921 % fragmented: 84.75 clusters: 10000 extents: 8475 score: 21696
Inode: 16922 % fragmented: 95.50 clusters: 10000 extents: 9550 score: 24448
resv_level=5 (defaults changed a bit, this means '128 blocks per reservation'):
Inode: 16916 % fragmented: 1.66 clusters: 10000 extents: 166 score: 425
Inode: 16917 % fragmented: 1.71 clusters: 10000 extents: 171 score: 438
Inode: 16918 % fragmented: 1.58 clusters: 10000 extents: 158 score: 404
Thanks in advance,
--Mark
PATCH 1/5 ocfs2: allocation reservations
This patch improves Ocfs2 allocation policy by allowing an inode to reserve a portion of the local alloc bitmap for itself. The reserved portion (allocation window) is advisory in that other allocation windows might steal it if the local alloc bitmap becomes full. Otherwise, the reservations are honored and guaranteed to be free. When the local alloc window is moved to a different portion of the bitmap, existing reservations are discarded.
Reservation windows are represented internally by a red-black tree. Within that tree, each node represents the reservation window of one inode. An LRU of active reservations is also maintained. When new data is written, we allocate it from the inodes window. When all bits in a window are exhausted, we allocate a new one as close to the previous one as possible. Should we not find free space, an existing reservation is pulled off the LRU and cannibalized.
--- Documentation/filesystems/ocfs2.txt | 3 + fs/ocfs2/Makefile | 1 + fs/ocfs2/cluster/masklog.c | 1 + fs/ocfs2/cluster/masklog.h | 1 + fs/ocfs2/localalloc.c | 64 +++- fs/ocfs2/ocfs2.h | 5 + fs/ocfs2/reservations.c | 829 +++++++++++++++++++++++++++++++++++ fs/ocfs2/reservations.h | 154 +++++++ fs/ocfs2/suballoc.h | 2 + fs/ocfs2/super.c | 25 + 10 files changed, 1074 insertions(+), 11 deletions(-) create mode 100644 fs/ocfs2/reservations.c create mode 100644 fs/ocfs2/reservations.h
Documentation/filesystems/ocfs2.txt +3 -0
@@ -80,3 +80,6 @@ user_xattr (*) Enables Extended User Attributes. nouser_xattr Disables Extended User Attributes. acl Enables POSIX Access Control Lists support. noacl (*) Disables POSIX Access Control Lists support. +resv_level=4 (*) Set how agressive allocation reservations will be. + Valid values are between 0 (reservations off) to 8 + (maximum space for reservations).
fs/ocfs2/Makefile +1 -0
@@ -29,6 +29,7 @@ ocfs2-objs := \ mmap.o \ namei.o \ refcounttree.o \ + reservations.o \ resize.o \ slot_map.o \ suballoc.o \
fs/ocfs2/cluster/masklog.c +1 -0
@@ -115,6 +115,7 @@ static struct mlog_attribute mlog_attrs[MLOG_MAX_BITS] = {define_mask (ERROR ),define_mask (NOTICE ),define_mask (KTHREAD ), + define_mask(RESERVATIONS), };static struct attribute *mlog_attr_ptrs[MLOG_MAX_BITS ] = {NULL , };
fs/ocfs2/cluster/masklog.h +1 -0
@@ -118,6 +118,7 @@#define ML_ERROR 0x0000000100000000ULL/* sent to KERN_ERR */ #define ML_NOTICE 0x0000000200000000ULL/* setn to KERN_NOTICE */ #define ML_KTHREAD 0x0000000400000000ULL/* kernel thread activity */ +#define ML_RESERVATIONS 0x0000000800000000ULL/* ocfs2 alloc reservations */ #define MLOG_INITIAL_AND_MASK (ML_ERROR|ML_NOTICE)#define MLOG_INITIAL_NOT_MASK (ML_ENTRY|ML_EXIT)
fs/ocfs2/localalloc.c +53 -11
138 lines not on this page · raw message
fs/ocfs2/ocfs2.h +5 -0
@@ -46,6 +46,7 @@/* For struct ocfs2_blockcheck_stats */ #include "blockcheck.h" +#include "reservations.h" /* Caching of metadata buffers */
@@ -346,6 +347,10 @@ struct ocfs2_superu64 la_last_gd; +struct ocfs2_reservation_map osb_la_resmap; + +unsigned int osb_resv_level; +/* Next three fields are for local node slot recovery during * mount. */ int dirty;
fs/ocfs2/reservations.c new file +829 -0
829 lines not on this page · raw message
fs/ocfs2/reservations.h new file +154 -0
154 lines not on this page · raw message
fs/ocfs2/suballoc.h +2 -0
@@ -54,6 +54,8 @@ struct ocfs2_alloc_context {u64 ac_last_group;u64 ac_max_block;/* Highest block number to allocate. 0 is is the same as ~0 - unlimited */ + +struct ocfs2_alloc_reservation *ac_resv; };void ocfs2_free_alloc_context (struct ocfs2_alloc_context *ac);
fs/ocfs2/super.c +25 -0
@@ -94,6 +94,7 @@ struct mount_optionsunsigned int atime_quantum;signed short slot;unsigned int localalloc_opt; +unsigned int resv_level;char cluster_stack[OCFS2_STACK_LABEL_LEN +1 ]; };
@@ -175,6 +176,7 @@ enum { Opt_noacl, Opt_usrquota, Opt_grpquota, + Opt_resv_level, Opt_err, };
@@ -201,6 +203,7 @@ static const match_table_t tokens = { {Opt_noacl, "noacl"}, {Opt_usrquota, "usrquota"}, {Opt_grpquota, "grpquota"}, + {Opt_resv_level, "resv_level=%u"}, {Opt_err, NULL} };
@@ -1026,6 +1029,7 @@ static int ocfs2_fill_super(struct super_block *sb, void *data, int silent) osb->osb_commit_interval = parsed_options.commit_interval; osb->local_alloc_default_bits =ocfs2_megabytes_to_clusters (sb, parsed_options.localalloc_opt); osb->local_alloc_bits = osb->local_alloc_default_bits; + osb->osb_resv_level = parsed_options.resv_level; status =ocfs2_verify_userspace_stack (osb, &parsed_options);if (status)
@@ -1286,6 +1290,7 @@ static int ocfs2_parse_options(struct super_block *sb, mopt->slot =OCFS2_INVALID_SLOT ; mopt->localalloc_opt =OCFS2_DEFAULT_LOCAL_ALLOC_SIZE ; mopt->cluster_stack[0 ] ='\0' ; + mopt->resv_level =OCFS2_DEFAULT_RESV_LEVEL ;if (!options) { status =1 ;
@@ -1429,6 +1434,17 @@ static int ocfs2_parse_options(struct super_block *sb, mopt->mount_opt |=OCFS2_MOUNT_NO_POSIX_ACL ; mopt->mount_opt &= ~OCFS2_MOUNT_POSIX_ACL ;break ; +case Opt_resv_level: +if (is_remount) +break ; +if (match_int (&args[0 ], &option)) { + status =0 ; + goto bail; + } +if (option >=OCFS2_MIN_RESV_LEVEL && + option <OCFS2_MAX_RESV_LEVEL ) + mopt->resv_level = option; +break ;default :mlog (ML_ERROR , "Unrecognized mount option \"%s\" "
@@ -1510,6 +1526,9 @@ static int ocfs2_show_options(struct seq_file *s, struct vfsmount *mnt)else seq_printf (s,",noacl" ); +if (osb->osb_resv_level !=OCFS2_DEFAULT_RESV_LEVEL ) +seq_printf (s,",resv_level=%d" , osb->osb_resv_level); +return 0 ; }
@@ -2038,6 +2057,12 @@ static int ocfs2_initialize_super(struct super_block *sb,init_waitqueue_head (&osb->osb_mount_event); + status =ocfs2_resmap_init (osb, &osb->osb_la_resmap); +if (status) { +mlog_errno (status); + goto bail; + } + osb->vol_label =kmalloc (OCFS2_MAX_VOL_LABEL_LEN ,GFP_KERNEL );if (!osb->vol_label) {mlog (ML_ERROR ,"unable to alloc vol label\n" );
Signature · 1 line
1.6.4.2
PATCH 1/5 ocfs2: allocation reservations
Quoted 3 lines · Mark Fasheh
Reservation windows are represented internally by a red-black tree. Within that tree, each node represents the reservation window of one inode. An LRU of active reservations is also maintained. When new
I see you decided against chopping up the localalloc and went
for reservations as needed.
fs/ocfs2/reservations.c line 210
Quoted 12 lines
fs/ocfs2/reservations.c new file +12
-0+int ocfs2_resmap_init (struct ocfs2_super *osb, +struct ocfs2_reservation_map *resmap) +{ +memset (resmap,0 ,sizeof (*resmap)); + + resmap->m_osb = osb; + resmap->m_reservations =RB_ROOT ; +/* m_bitmap_len is initialized to zero by the above memset. */ +INIT_LIST_HEAD (&resmap->m_lru); + +return 0 ; +}
Why have m_osb and set it? Can't you get the osb fromcontainer_of(resmap, struct ocfs2_super, osb_la_resmap)?
fs/ocfs2/reservations.c line 708
Quoted 3 lines
fs/ocfs2/reservations.c new file +3
-0+int ocfs2_resmap_resv_bits (struct ocfs2_reservation_map *resmap, +struct ocfs2_alloc_reservation *resv, +int tmpwindow,int *cstart,int *clen)
The tmpwindow argument to resmap_resv_bits()...what you really
want to say is "I'm not interested in hanging on to reserved space. Please behave like the original localalloc". What it does is correct. I wish I had a better way to say it.
Quoted 16 lines
fs/ocfs2/reservations.h new file +13
-0@@ -0,0 +1,154 @@ +/* -*- mode: c; c-basic-offset: 8; -*- +* vim: noexpandtab sw=8 ts=8 sts=0: +* +* reservations.h +* +* Allocation reservations function prototypes and structures. +* +* Copyright (C) 2009 Novell. All rights reserved. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public +* License as published by the Free Software Foundation; either +* version 2 of the License, or (at your option) any later version.
fs/ocfs2/reservations.h line 23
Quoted 10 lines
fs/ocfs2/reservations.h new file +10
-0+* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; if not, write to the +* Free Software Foundation, Inc., 59 Temple Place - Suite 330, +* Boston, MA 021110-1307, USA.
Don't add the paragraph about Temple Place to new files ;-)
Joel
Signature · 10 lines
"Always give your best, never get discouraged, never be petty; always remember, others may hate you. Those who hate you don't win unless you hate them. And then you destroy yourself." - Richard M. Nixon Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 1/5 ocfs2: allocation reservations
Quoted 7 lines · Joel Becker
Quoted 3 lines · Mark Fasheh
Reservation windows are represented internally by a red-black tree. Within that tree, each node represents the reservation window of one inode. An LRU of active reservations is also maintained. When new
I see you decided against chopping up the localalloc and went
for reservations as needed.
Yeah, in the end it wound up making the most sense for our usage. In reality though, the bitmap is being "virtually" chopped up.
Quoted 7 lines
Quoted 7 lines
fs/ocfs2/reservations.c new file +7
-0+int ocfs2_resmap_init (struct ocfs2_super *osb, +struct ocfs2_reservation_map *resmap) +{ +memset (resmap,0 ,sizeof (*resmap)); + + resmap->m_osb = osb; + resmap->m_reservations =RB_ROOT ;
Quoted 8 lines
fs/ocfs2/reservations.c line 210
Quoted 5 lines
fs/ocfs2/reservations.c new file +5
-0+/* m_bitmap_len is initialized to zero by the above memset. */ +INIT_LIST_HEAD (&resmap->m_lru); + +return 0 ; +}Why have
m_osband set it? Can't you get the osb fromcontainer_of(resmap, struct ocfs2_super, osb_la_resmap)?
I could, but I have m_osb there to divorce it from any particular parent structure. That way a future version to use this on say, block groups wouldn't need any changes in reservations.[ch].
Quoted 8 lines
fs/ocfs2/reservations.c line 708
Quoted 3 lines
fs/ocfs2/reservations.c new file +3
-0+int ocfs2_resmap_resv_bits (struct ocfs2_reservation_map *resmap, +struct ocfs2_alloc_reservation *resv, +int tmpwindow,int *cstart,int *clen)The tmpwindow argument to resmap_resv_bits()...what you really
want to say is "I'm not interested in hanging on to reserved space. Please behave like the original localalloc". What it does is correct. I wish I had a better way to say it.
Essentially that's what happens, but with the
benefits of allocating the very small window in the right places, etc. Otherwise we'd be searching the bitmap without tracking other reservations, etc.
Quoted 20 lines
Quoted 20 lines
fs/ocfs2/reservations.h new file +17
-0@@ -0,0 +1,154 @@ +/* -*- mode: c; c-basic-offset: 8; -*- +* vim: noexpandtab sw=8 ts=8 sts=0: +* +* reservations.h +* +* Allocation reservations function prototypes and structures. +* +* Copyright (C) 2009 Novell. All rights reserved. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public +* License as published by the Free Software Foundation; either +* version 2 of the License, or (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Quoted 8 lines
fs/ocfs2/reservations.h line 23
Quoted 6 lines
fs/ocfs2/reservations.h new file +6
-0+* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; if not, write to the +* Free Software Foundation, Inc., 59 Temple Place - Suite 330, +* Boston, MA 021110-1307, USA. Don't add the paragraph about Temple Place to new files ;-)
Doh! I thought I had already done that in the last round. I'll fix it up.
--Mark
--
Mark Fasheh
PATCH 2/5 ocfs2: use allocation reservations during file write
In reply to Mark FashehAdd a per-inode reservations structure and pass it through to the reservations code.
--- fs/ocfs2/alloc.c | 2 ++ fs/ocfs2/aops.c | 3 +++ fs/ocfs2/file.c | 3 +++ fs/ocfs2/inode.c | 4 ++++ fs/ocfs2/inode.h | 2 ++ fs/ocfs2/super.c | 2 ++ 6 files changed, 16 insertions(+), 0 deletions(-)
fs/ocfs2/alloc.c +2 -0
@@ -7307,6 +7307,8 @@ int ocfs2_convert_inline_data_to_extents(struct inode *inode, } did_quota =1 ; + data_ac->ac_resv = &OCFS2_I (inode)->ip_la_data_resv; + ret =ocfs2_claim_clusters (osb, handle, data_ac,1 , &bit_off, &num);if (ret) {
fs/ocfs2/aops.c +3 -0
@@ -1734,6 +1734,9 @@ int ocfs2_write_begin_nolock(struct address_space *mapping, goto out; } +if (data_ac) + data_ac->ac_resv = &OCFS2_I (inode)->ip_la_data_resv; + credits =ocfs2_calc_extend_credits (inode->i_sb, &di->id2.i_list, clusters_to_alloc);
fs/ocfs2/file.c +3 -0
@@ -485,6 +485,9 @@ static int ocfs2_truncate_file(struct inode *inode,down_write (&OCFS2_I (inode)->ip_alloc_sem); +ocfs2_resv_discard (&osb->osb_la_resmap, + &OCFS2_I (inode)->ip_la_data_resv); +/* * The inode lock forced other nodes to sync and drop their * pages, which (correctly) happens even if we have a truncate
fs/ocfs2/inode.c +4 -0
@@ -1097,6 +1097,10 @@ void ocfs2_clear_inode(struct inode *inode)ocfs2_mark_lockres_freeing (&oi->ip_inode_lockres);ocfs2_mark_lockres_freeing (&oi->ip_open_lockres); +ocfs2_resv_discard (&OCFS2_SB (inode->i_sb)->osb_la_resmap, + &oi->ip_la_data_resv); +ocfs2_resv_init_once (&oi->ip_la_data_resv); +/* We very well may get a clear_inode before all an inodes * metadata has hit disk. Of course, we can't drop any cluster * locks until the journal has finished with it. The only
fs/ocfs2/inode.h +2 -0
@@ -70,6 +70,8 @@ struct ocfs2_inode_info/* Only valid if the inode is the dir. */ u32 ip_last_used_slot;u64 ip_last_used_group; + +struct ocfs2_alloc_reservation ip_la_data_resv; };/*
fs/ocfs2/super.c +2 -0
@@ -1703,6 +1703,8 @@ static void ocfs2_inode_init_once(void *data) oi->ip_blkno =0ULL ; oi->ip_clusters =0 ; +ocfs2_resv_init_once (&oi->ip_la_data_resv); +ocfs2_lock_res_init_once (&oi->ip_rw_lockres);ocfs2_lock_res_init_once (&oi->ip_inode_lockres);ocfs2_lock_res_init_once (&oi->ip_open_lockres);
Signature · 1 line
1.6.4.2
PATCH 3/5 ocfs2: use allocation reservations for directory data
In reply to Mark FashehUse the reservations system for unindexed dir tree allocations. We don't bother with the indexed tree as reads from it are mostly random anyway. By default this behavior is turned off and can be turned on via mount option 'dir_resv'. Workloads which create many large directories will want to turn it on.
A future improvement will be to use a different window size for directory inodes. Once done, we should change the default for this feature to be enabled.
--- Documentation/filesystems/ocfs2.txt | 2 ++ fs/ocfs2/dir.c | 4 ++++ fs/ocfs2/ocfs2.h | 2 ++ fs/ocfs2/suballoc.c | 1 + fs/ocfs2/super.c | 15 +++++++++++++++ 5 files changed, 24 insertions(+), 0 deletions(-)
Documentation/filesystems/ocfs2.txt +2 -0
@@ -83,3 +83,5 @@ noacl (*) Disables POSIX Access Control Lists support. resv_level=4 (*) Set how agressive allocation reservations will be. Valid values are between 0 (reservations off) to 8 (maximum space for reservations). +no_dir_resv (*) Don't get allocation reservations on directory inodes. +dir_resv Get allocation reservations on directory inodes.
fs/ocfs2/dir.c +4 -0
@@ -2993,6 +2993,8 @@ static int ocfs2_expand_inline_dir(struct inode *dir, struct buffer_head *di_bh,* if we only get one now, that's enough to continue. The rest * will be claimed after the conversion to extents. */ +if (osb->s_mount_opt &OCFS2_MOUNT_DIR_RESV ) + data_ac->ac_resv = &oi->ip_la_data_resv; ret =ocfs2_claim_clusters (osb, handle, data_ac,1 , &bit_off, &len);if (ret) {mlog_errno (ret);
@@ -3371,6 +3373,8 @@ static int ocfs2_extend_dir(struct ocfs2_super *osb,mlog_errno (status); goto bail; } +if (osb->s_mount_opt &OCFS2_MOUNT_DIR_RESV ) + data_ac->ac_resv = &OCFS2_I (dir)->ip_la_data_resv; credits =ocfs2_calc_extend_credits (sb, el,1 ); }else {
fs/ocfs2/ocfs2.h +2 -0
@@ -255,6 +255,8 @@ enum ocfs2_mount_optionscontrol lists */ OCFS2_MOUNT_USRQUOTA =1 <<10 ,/* We support user quotas */ OCFS2_MOUNT_GRPQUOTA =1 <<11 ,/* We support group quotas */ + OCFS2_MOUNT_DIR_RESV = 1 << 12, /* Get reservations + * on directories */ };#define OCFS2_OSB_SOFT_RO 0x0001
fs/ocfs2/suballoc.c +1 -0
@@ -137,6 +137,7 @@ void ocfs2_free_ac_resource(struct ocfs2_alloc_context *ac) }brelse (ac->ac_bh); ac->ac_bh =NULL ; + ac->ac_resv =NULL ; }void ocfs2_free_alloc_context (struct ocfs2_alloc_context *ac)
fs/ocfs2/super.c +15 -0
@@ -177,6 +177,8 @@ enum { Opt_usrquota, Opt_grpquota, Opt_resv_level, + Opt_dir_resv, + Opt_no_dir_resv, Opt_err, };
@@ -204,6 +206,8 @@ static const match_table_t tokens = { {Opt_usrquota, "usrquota"}, {Opt_grpquota, "grpquota"}, {Opt_resv_level, "resv_level=%u"}, + {Opt_dir_resv, "dir_resv"}, + {Opt_no_dir_resv, "no_dir_resv"}, {Opt_err, NULL} };
@@ -1445,6 +1449,12 @@ static int ocfs2_parse_options(struct super_block *sb, option < OCFS2_MAX_RESV_LEVEL) mopt->resv_level = option;break ; +case Opt_dir_resv: + mopt->mount_opt |=OCFS2_MOUNT_DIR_RESV ; +break ; +case Opt_no_dir_resv: + mopt->mount_opt &= ~OCFS2_MOUNT_DIR_RESV ; +break ;default :mlog (ML_ERROR , "Unrecognized mount option \"%s\" "
@@ -1529,6 +1539,11 @@ static int ocfs2_show_options(struct seq_file *s, struct vfsmount *mnt)if (osb->osb_resv_level !=OCFS2_DEFAULT_RESV_LEVEL )seq_printf (s,",resv_level=%d" , osb->osb_resv_level); +if (opts &OCFS2_MOUNT_DIR_RESV ) +seq_printf (s,",dir_resv" ); +else +seq_printf (s,",no_dir_resv" ); +return 0 ; }
Signature · 1 line
1.6.4.2
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 9 lines · Mark Fasheh
Use the reservations system for unindexed dir tree allocations. We don't bother with the indexed tree as reads from it are mostly random anyway. By default this behavior is turned off and can be turned on via mount option '
dir_resv'. Workloads which create many large directories will want to turn it on.A future improvement will be to use a different window size for directory inodes. Once done, we should change the default for this feature to be enabled.
Why can't we turn this on and avoid a mount option? Sure, the
reservation holds out some space, but it will be cannibalized if the directory doesn't grow, right?
Joel
Signature · 9 lines
"I inject pure kryptonite into my brain. It improves my kung fu, and it eases the pain." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 7 lines · Joel Becker
Quoted 6 lines · Mark Fasheh
Use the reservations system for unindexed dir tree allocations. We don't bother with the indexed tree as reads from it are mostly random anyway. By default this behavior is turned off and can be turned on via mount option '
dir_resv'. Workloads which create many large directories will want to turn it on.
Quoted 7 lines
Quoted 3 lines
A future improvement will be to use a different window size for directory inodes. Once done, we should change the default for this feature to be enabled.
Why can't we turn this on and avoid a mount option? Sure, the
reservation holds out some space, but it will be cannibalized if the directory doesn't grow, right?
Yeah, it'll work. My concern was that we'd be cannibalizing those too much since the window sizes are optimized for file data. It's only a hunch though that directories might want smaller windows - I didn't get much data on directory growth since my focus was on file data.
Maybe I should just go ahead and hack it up to know if a reservation is for a directory and it could provide some fewer number of bits (maybe half the default)? Adjusting the window size values is something we can do very easily until we get it right.
Adding a flag field on the resv struct could accomplish this pretty easily, and I could re-use the flag for the 'temporary' reservations, thus losing a function call parameter in the process.
I agree though that losing the extra mount option would be good.
--Mark
--
Mark Fasheh
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 4 lines · Mark Fasheh
Yeah, it'll work. My concern was that we'd be cannibalizing those too much since the window sizes are optimized for file data. It's only a hunch though that directories might want smaller windows - I didn't get much data on directory growth since my focus was on file data.
I expect that directories won't use their entire reservation.
I'm just not sure it matters. Half of the files we create won't either. If we actually are doing a lot of creating (untar, etc), we'll eventually canabalize the reservation attached to the directory anyway. So I don't know why directories have to have smaller reservations. Just let the expire/canabalize code handle it.
I'd be interested to see how an untar of a kernel tree or any
long-running workload that is helped by reservations changes when a directory is reserving the same space as a file.
Joel
Signature · 7 lines
"War doesn't determine who's right; war determines who's left." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 6 lines · Joel Becker
Quoted 4 lines · Mark Fasheh
Yeah, it'll work. My concern was that we'd be cannibalizing those too much since the window sizes are optimized for file data. It's only a hunch though that directories might want smaller windows - I didn't get much data on directory growth since my focus was on file data.
Quoted 6 lines
I expect that directories won't use their entire reservation.
I'm just not sure it matters. Half of the files we create won't either. If we actually are doing a lot of creating (untar, etc), we'll eventually canabalize the reservation attached to the directory anyway. So I don't know why directories have to have smaller reservations. Just let the expire/canabalize code handle it.
Cannibalizing a reservation is a bit less optimal because you're taking any region at that point, as opposed to trying very hard to be at least after the previous window. Also there's the shrinking logic (though in the last couple days I've been thinking of getting rid of region shrinking).
Anyway, I'm with you regarding what the proper parameters for directory reservations are. The fact is, i don't really know one way or the other. A mount option seemed to at least give the user an 'out' if things go bad.
Quoted 3 lines
I'd be interested to see how an untar of a kernel tree or any
long-running workload that is helped by reservations changes when a directory is reserving the same space as a file.
Well, we could gather up some disk images to get us that information. The patches as they are now make it easy to turn directory reservations on and off, so running some tests doesn't even need a reboot :)
I'm not sure that untar of a kernel tree will give us anything interesting. A parallel build would work... We could check a few arbitrary object files to see where they're at. Those tend to be skew bigger.
--Mark
--
Mark Fasheh
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 3 lines · Mark Fasheh
Anyway, I'm with you regarding what the proper parameters for directory reservations are. The fact is, i don't really know one way or the other. A mount option seemed to at least give the user an 'out' if things go bad.
Yeah, I hear you. I just figure we're stuck with the option for
the future.
Quoted 3 lines
I'm not sure that untar of a kernel tree will give us anything interesting. A parallel build would work... We could check a few arbitrary object files to see where they're at. Those tend to be skew bigger.
I was thinking about a kernel tree untar where the directories
are holding on to reservations as thousands of files are created under them. Wouldn't that lead to cannibalization and show us that pattern? This is just a lay guess - you have a lot more familiarity with the code.
Joel
Signature · 9 lines
"In a crisis, don't hide behind anything or anybody. They're going to find you anyway." - Paul "Bear" Bryant Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 7 lines · Joel Becker
Quoted 3 lines · Mark Fasheh
Anyway, I'm with you regarding what the proper parameters for directory reservations are. The fact is, i don't really know one way or the other. A mount option seemed to at least give the user an 'out' if things go bad.
Yeah, I hear you. I just figure we're stuck with the option for
the future.
Updated patches are up in git:
git pull git://git.kernel.org/pub/scm/linux/kernel/git/mfasheh/ocfs2-mark.git reservations
Aside from cleaning up the license info, I remove the dir_resv/no_dir_resv options. I also added a flags field to struct ocfs2_alloc_reservation which saves us the 'tmpwindow' argument to ocfs2_resmap_resv_bits().
Quoted 9 lines
Quoted 3 lines
I'm not sure that untar of a kernel tree will give us anything interesting. A parallel build would work... We could check a few arbitrary object files to see where they're at. Those tend to be skew bigger.
I was thinking about a kernel tree untar where the directories
are holding on to reservations as thousands of files are created under them. Wouldn't that lead to cannibalization and show us that pattern? This is just a lay guess - you have a lot more familiarity with the code.
I don't know how much a kernel build is going to make a difference. I took a couple images after kernel builds with various options but didn't see anything obvious. To be fair though, I only checked one or two files. I'll upload the images somewhere shortly.
As an interim compromise, I changed the code to get minimum (8 bits) sized windows on directories. That way, they'll get some amount of continguousness, but not as much as file data. We can easily adjust in any direction we want.
--Mark
--
Mark Fasheh
PATCH 3/5 ocfs2: use allocation reservations for directory data
Quoted 5 lines · Mark Fasheh
On Fri, Mar 19, 2010 at 11:18:48PM -0700, Joel Becker wrote:
Aside from cleaning up the license info, I remove the dir_resv/no_dir_resv options. I also added a flags field to struct
ocfs2_alloc_reservationwhich saves us the 'tmpwindow' argument toocfs2_resmap_resv_bits().
I like the changes.
Quoted 11 lines
Quoted 5 lines
I was thinking about a kernel tree untar where the directories
are holding on to reservations as thousands of files are created under them. Wouldn't that lead to cannibalization and show us that pattern? This is just a lay guess - you have a lot more familiarity with the code.I don't know how much a kernel build is going to make a difference. I took a couple images after kernel builds with various options but didn't see anything obvious. To be fair though, I only checked one or two files. I'll upload the images somewhere shortly.
Quoted 4 lines
As an interim compromise, I changed the code to get minimum (8 bits) sized windows on directories. That way, they'll get some amount of continguousness, but not as much as file data. We can easily adjust in any direction we want.
I wonder how the varying reservation sizes will impact
ulilization of the localalloc. May not matter, but there's a part of me that wonders. Does the new dir-resv code do as well on your tests as the no-dir-resv code?
Maybe something to see any help from dir reservations would be a
many-parallel-untar sort of thing? Unpack 20 kernel trees at the same time? Some other package with a lot of files. I have no idea if this will mean anything, and I actually rather trust your observations of previous untars. I'm just casting about for something to show us how dir reservations are behaving.
Joel
Signature · 9 lines
Life's Little Instruction Book #451 "Don't be afraid to say, 'I'm sorry.'" Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 4/5 ocfs2: allocate btree internal block groups from the global bitmap
In reply to Mark FashehOtherwise, the need for a very large contiguous allocation tends to wreak havoc on many inode allocation reservations on the local alloc, thus ruining any chances for contiguousness.
--- fs/ocfs2/suballoc.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
fs/ocfs2/suballoc.c +4 -1
@@ -659,7 +659,8 @@ int ocfs2_reserve_new_metadata_blocks(struct ocfs2_super *osb, status =ocfs2_reserve_suballoc_bits (osb, (*ac),EXTENT_ALLOC_SYSTEM_INODE , - slot,NULL ,ALLOC_NEW_GROUP ); + slot,NULL , +ALLOC_GROUPS_FROM_GLOBAL |ALLOC_NEW_GROUP );if (status <0 ) {if (status != -ENOSPC )mlog_errno (status);
@@ -1821,6 +1822,8 @@ int __ocfs2_claim_clusters(struct ocfs2_super *osb, && ac->ac_which != OCFS2_AC_USE_MAIN);if (ac->ac_which ==OCFS2_AC_USE_LOCAL ) { +WARN_ON (min_clusters >1 ); + status = ocfs2_claim_local_alloc_bits(osb, handle, ac,
Signature · 1 line
1.6.4.2
PATCH 5/5 ocfs2: remove ocfs2_local_alloc_in_range()
In reply to Mark FashehInodes are always allocated from the global bitmap now so we don't need this any more. Also, the existing implementation bounces reservations around needlessly.
--- fs/ocfs2/localalloc.c | 51 ------------------------------------------------- fs/ocfs2/suballoc.c | 6 +---- 2 files changed, 1 insertions(+), 56 deletions(-)
fs/ocfs2/localalloc.c +0 -51
@@ -484,46 +484,6 @@ out:return status; } -/* Check to see if the local alloc window is within ac->ac_max_block */ -static int ocfs2_local_alloc_in_range (struct inode *inode, -struct ocfs2_alloc_context *ac, -u32 bits_wanted) -{ -struct ocfs2_super *osb =OCFS2_SB (inode->i_sb); -struct ocfs2_dinode *alloc; -struct ocfs2_local_alloc *la; -int start; -u64 block_off; - -if (!ac->ac_max_block) -return 1 ; - - alloc = (struct ocfs2_dinode *) osb->local_alloc_bh->b_data; - la =OCFS2_LOCAL_ALLOC (alloc); - - start =ocfs2_local_alloc_find_clear_bits (osb, alloc, &bits_wanted,NULL ); -if (start ==-1 ) { -mlog_errno (-ENOSPC ); -return 0 ; - } - -/* -* Converting (bm_off + start + bits_wanted) to blocks gives us -* the blkno just past our actual allocation. This is perfect -* to compare with ac_max_block. -*/ - block_off =ocfs2_clusters_to_blocks (inode->i_sb, -le32_to_cpu (la->la_bm_off) + - start + bits_wanted); -mlog (0 ,"Checking %llu against %llu\n" , - (unsigned long long )block_off, - (unsigned long long )ac->ac_max_block); -if (block_off > ac->ac_max_block) -return 0 ; - -return 1 ; -} -/* * make sure we've got at least bits_wanted contiguous bits in the * local alloc. You lose them when you drop i_mutex.
@@ -616,17 +576,6 @@ int ocfs2_reserve_local_alloc_bits(struct ocfs2_super *osb,mlog (0 ,"Calling in_range for max block %llu\n" , (unsigned long long )ac->ac_max_block); -if (!ocfs2_local_alloc_in_range (local_alloc_inode, ac, - bits_wanted)) { -/* -* The window is outside ac->ac_max_block. -* This errno tells the caller to keep localalloc enabled -* but to get the allocation from the main bitmap. -*/ - status = -EFBIG ; - goto bail; - } - ac->ac_inode = local_alloc_inode;/* We should never use localalloc from another slot */ ac->ac_alloc_slot = osb->slot_num;
fs/ocfs2/suballoc.c +1 -5
@@ -861,11 +861,7 @@ static int ocfs2_reserve_clusters_with_limit(struct ocfs2_super *osb, status =ocfs2_reserve_local_alloc_bits (osb, bits_wanted, *ac); -if (status == -EFBIG ) { -/* The local alloc window is outside ac_max_block. -* use the main bitmap. */ - status = -ENOSPC ; - }else if ((status <0 ) && (status != -ENOSPC )) { +if ((status <0 ) && (status != -ENOSPC )) {mlog_errno (status); goto bail; }
Signature · 1 line
1.6.4.2
cover Ocfs2 allocation reservations
In reply to Mark FashehQuoted 14 lines · Mark Fasheh
Changes from the last patch set:
- added a check for overlapping reservations inocfs2_resv_insert()
- cleaned up the comments inocfs2_cannibalize_resv()
- the check for reservation past bitmap end inocfs2_check_resmap()is more strict now.
- removed the unusedm_search_startmember ofocfs2_reservation_map
- optimized__ocfs2_resv_find_window()to ignore regions that are too small for the current alloc
- major cleanup ofocfs2_resmap_claimed_bits()
- added a set of BUG_ON's to inocfs2_resmap_claimed_bits()to check that the passed allocation range is within the window.
- fixedocfs2_local_alloc_find_clear_bits()to return actual bits allocated
- add a check for a nulldata_acinocfs2_write_begin_nolock()
Quoted 2 lines
I also added a fifth patch, "ocfs2: remove
ocfs2_local_alloc_in_range()". I could spin this as it's own patch to go upstream earlier if we want.
Attached is a very small follow-up patch to turn off some of the expensive debug checks I placed in the code.
--Mark
From: Mark Fasheh <[email protected]>
[PATCH] ocfs2: turn off expensive reservations debugging by default
They can still be turned back on by enabling the "expensive checks" config option for Ocfs2.
--- fs/ocfs2/reservations.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-)
fs/ocfs2/reservations.c +0 -3
@@ -41,9 +41,6 @@#define OCFS2_CHECK_RESERVATIONS #endif -#define OCFS2_CHECK_RESERVATIONS - -DEFINE_SPINLOCK (resv_lock);#define OCFS2_MIN_RESV_WINDOW_BITS 8
Signature · 1 line
1.6.4.2