Skip to contentSkip to conversation

Ocfs2 allocation reservations

RFC New Mark Fasheh 2 participants 7 messages Dec 15, 2009 – Jan 20, 2010

Subsystems (2)
Documentation OCFS2
lore
Share & tools
lore

Mark Fasheh

author
#0 lore raw

cover Ocfs2 allocation reservations

Hi, the following patches implement allocation reservations for Ocfs2. Right now I'd consider them to be beta quality - simple operations work but the code likely won't survive a kernel build. As it is implemented, the code only takes reservations on the local alloc bitmap. I've divorced the structures from any bitmap type however, so we can extend this to the chain allocators with only a small amount of code. Also, there's nothing in reservations.c which depends on a struct inode.

Also worth mentioning is that the mount option ("resv_level=") was kind of hastily put together. I think we definitely want the option to control how aggressive the code is with reservations, but I'm not sure I'm happy with the implementation of that knob.

I still need to do some tuning, but already an increase in contiguousness can be shown in this small test:

/root/extendo /ocfs2/try1 4 1000 32 & \
/root/extendo /ocfs2/try2 4 1000 32 & \
/root/extendo /ocfs2/try3 4 1000 32 &

So that runs 3 instances of 'extendo' (from ocfs2-test) which should write about 4k 32 times, with a 1 second wait in between runs. Test file system was formatted with 4k clusters and inline-data turned off.

Without reservations turned on, we get perfect fragmentation.

Inode: 66007 % fragmented: 100.00 clusters: 32 extents: 32
Inode: 66009 % fragmented: 100.00 clusters: 32 extents: 32
Inode: 66008 % fragmented: 100.00 clusters: 32 extents: 32

This version of the reservations code shows the following improvements:

Inode: 66007 % fragmented: 50.00 clusters: 32 extents: 16
Inode: 66009 % fragmented: 68.75 clusters: 32 extents: 22
Inode: 66008 % fragmented: 43.75 clusters: 32 extents: 14

All comments are appreciated.

	--Mark

--
Mark Fasheh

author
#1 lore raw

RFC 1/3 ocfs2: allocation reservations

This patch improves Ocfs2 allocation policy by allowing an inode to reserve a portion of the local alloc bitmap for itself. Allocation windows are advisory in that they won't block use of that portion of the bitmap. This makes dealing with corner cases much easier - we can always fall back to previous policy.

Reservation windows are represented internally by a red-black tree. Within that tree, each node represents the reservation window of one inode. When new data is written, we try to allocate from the window first. If that allocation fails, we fall back to our old heuristics and a new window is computed from the results. Allocation windows will also be extended if allocation from them succeeds.

Signed-off-by Mark Fasheh <[email protected]>
---
 Documentation/filesystems/ocfs2.txt |    3 +
 fs/ocfs2/Makefile                   |    1 +
 fs/ocfs2/cluster/masklog.c          |    1 +
 fs/ocfs2/cluster/masklog.h          |    1 +
 fs/ocfs2/localalloc.c               |   39 ++-
 fs/ocfs2/ocfs2.h                    |    5 +
 fs/ocfs2/reservations.c             |  624 +++++++++++++++++++++++++++++++++++
 fs/ocfs2/reservations.h             |  149 +++++++++
 fs/ocfs2/suballoc.h                 |    2 +
 fs/ocfs2/super.c                    |   26 ++
 10 files changed, 845 insertions(+), 6 deletions(-)
 create mode 100644 fs/ocfs2/reservations.c
 create mode 100644 fs/ocfs2/reservations.h
Documentation/filesystems/ocfs2.txt +3 -0
index c58b9f5..0e94600 100644
@@ -80,3 +80,6 @@ user_xattr	(*)	Enables Extended User Attributes.
 nouser_xattr		Disables Extended User Attributes.
 acl			Enables POSIX Access Control Lists support.
 noacl		(*)	Disables POSIX Access Control Lists support.
+resv_level=3	(*)	Set how agressive allocation reservations will be.
+			Valid values are between 0 (reservations off) to 6
+			(maximum space for reservations).
fs/ocfs2/Makefile +1 -0
index 31f25ce..c8eee00 100644
@@ -29,6 +29,7 @@ ocfs2-objs := \
 	mmap.o 			\
 	namei.o 		\
 	refcounttree.o		\
+	reservations.o		\
 	resize.o		\
 	slot_map.o 		\
 	suballoc.o 		\
fs/ocfs2/cluster/masklog.c +1 -0
index 1cd2934..a56147a 100644
@@ -115,6 +115,7 @@ static struct mlog_attribute mlog_attrs[MLOG_MAX_BITS] = {
 	define_mask(ERROR),
 	define_mask(NOTICE),
 	define_mask(KTHREAD),
+	define_mask(RESERVATIONS),
 };
 
 static struct attribute *mlog_attr_ptrs[MLOG_MAX_BITS] = {NULL, };
fs/ocfs2/cluster/masklog.h +1 -0
index 9b4d117..af1a610 100644
@@ -118,6 +118,7 @@
 #define ML_ERROR	0x0000000100000000ULL /* sent to KERN_ERR */
 #define ML_NOTICE	0x0000000200000000ULL /* setn to KERN_NOTICE */
 #define ML_KTHREAD	0x0000000400000000ULL /* kernel thread activity */
+#define	ML_RESERVATIONS	0x0000000800000000ULL /* ocfs2 alloc reservations */
 
 #define MLOG_INITIAL_AND_MASK (ML_ERROR|ML_NOTICE)
 #define MLOG_INITIAL_NOT_MASK (ML_ENTRY|ML_EXIT)
fs/ocfs2/localalloc.c +33 -6
index ac10f83..2042680 100644
@@ -52,7 +52,8 @@ static u32 ocfs2_local_alloc_count_bits(struct ocfs2_dinode *alloc);
 
 static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
 					     struct ocfs2_dinode *alloc,
-					     u32 numbits);
+					     u32 numbits,
+					     struct ocfs2_alloc_reservation *resv);
 
 static void ocfs2_clear_local_alloc(struct ocfs2_dinode *alloc);
 

@@ -262,6 +263,8 @@ void ocfs2_shutdown_local_alloc(struct ocfs2_super *osb)
 
 	osb->local_alloc_state = OCFS2_LA_DISABLED;
 
+	ocfs2_resmap_uninit(&osb->osb_la_resmap);
+
 	main_bm_inode = ocfs2_get_system_file_inode(osb,
 						    GLOBAL_BITMAP_SYSTEM_INODE,
 						    OCFS2_INVALID_SLOT);

@@ -498,7 +501,7 @@ static int ocfs2_local_alloc_in_range(struct inode *inode,
 	alloc = (struct ocfs2_dinode *) osb->local_alloc_bh->b_data;
 	la = OCFS2_LOCAL_ALLOC(alloc);
 
-	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted);
+	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted, NULL);
 	if (start == -1) {
 		mlog_errno(-ENOSPC);
 		return 0;

@@ -664,7 +667,8 @@ int ocfs2_claim_local_alloc_bits(struct ocfs2_super *osb,
 	alloc = (struct ocfs2_dinode *) osb->local_alloc_bh->b_data;
 	la = OCFS2_LOCAL_ALLOC(alloc);
 
-	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted);
+	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted,
+						  ac->ac_resv);
 	if (start == -1) {
 		/* TODO: Shouldn't we just BUG here? */
 		status = -ENOSPC;

@@ -687,6 +691,9 @@ int ocfs2_claim_local_alloc_bits(struct ocfs2_super *osb,
 		goto bail;
 	}
 
+	ocfs2_resmap_claimed_bits(&osb->osb_la_resmap, ac->ac_resv, start,
+				  bits_wanted);
+
 	while(bits_wanted--)
 		ocfs2_set_bit(start++, bitmap);
 

@@ -722,11 +729,13 @@ static u32 ocfs2_local_alloc_count_bits(struct ocfs2_dinode *alloc)
 }
 
 static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
-					     struct ocfs2_dinode *alloc,
-					     u32 numbits)
+				     struct ocfs2_dinode *alloc,
+				     u32 numbits,
+				     struct ocfs2_alloc_reservation *resv)
 {
 	int numfound, bitoff, left, startoff, lastzero;
 	void *bitmap = NULL;
+	struct ocfs2_reservation_map *resmap = &osb->osb_la_resmap;
 
 	mlog_entry("(numbits wanted = %u)\n", numbits);
 

@@ -738,6 +747,20 @@ static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
 
 	bitmap = OCFS2_LOCAL_ALLOC(alloc)->la_bitmap;
 
+	/*
+	 * Ask the reservations code first whether this request can be
+	 * easily fulfilled. No errors here are fatal - if we didn't
+	 * find the number of bits needed, we'll just take the slow
+	 * path.
+	 */
+	if (ocfs2_resmap_resv_bits(resmap, resv, bitmap, &bitoff, &numfound)
+	    == 0) {
+		if (numfound >= numbits) {
+			numfound = numbits;
+			goto bail;
+		}
+	}
+
 	numfound = bitoff = startoff = 0;
 	lastzero = -1;
 	left = le32_to_cpu(alloc->id1.bitmap1.i_total);

@@ -772,8 +795,10 @@ static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
 
 	if (numfound == numbits)
 		bitoff = startoff - numfound;
-	else
+	else {
+		numfound = 0;
 		bitoff = -1;
+	}
 
 bail:
 	mlog_exit(bitoff);

@@ -1096,6 +1121,8 @@ retry_enospc:
 	memset(OCFS2_LOCAL_ALLOC(alloc)->la_bitmap, 0,
 	       le16_to_cpu(la->la_size));
 
+	ocfs2_resmap_restart(&osb->osb_la_resmap, cluster_count);
+
 	mlog(0, "New window allocated:\n");
 	mlog(0, "window la_bm_off = %u\n",
 	     OCFS2_LOCAL_ALLOC(alloc)->la_bm_off);
fs/ocfs2/ocfs2.h +5 -0
index d963d86..5a6300f 100644
@@ -46,6 +46,7 @@
 /* For struct ocfs2_blockcheck_stats */
 #include "blockcheck.h"
 
+#include "reservations.h"
 
 /* Caching of metadata buffers */
 

@@ -340,6 +341,10 @@ struct ocfs2_super
 
 	u64 la_last_gd;
 
+	struct ocfs2_reservation_map	osb_la_resmap;
+
+	unsigned int	osb_resv_level;
+
 	/* Next three fields are for local node slot recovery during
 	 * mount. */
 	int dirty;
fs/ocfs2/reservations.c new file +624 -0

624 lines not on this page · raw message

fs/ocfs2/reservations.h new file +149 -0

149 lines not on this page · raw message

fs/ocfs2/suballoc.h +2 -0
index 8c9a78a..5eb7753 100644
@@ -54,6 +54,8 @@ struct ocfs2_alloc_context {
 	u64    ac_last_group;
 	u64    ac_max_block;  /* Highest block number to allocate. 0 is
 				 is the same as ~0 - unlimited */
+
+	struct ocfs2_alloc_reservation	*ac_resv;
 };
 
 void ocfs2_free_alloc_context(struct ocfs2_alloc_context *ac);
fs/ocfs2/super.c +26 -0
index 14f47d2..62b46a2 100644
@@ -94,6 +94,7 @@ struct mount_options
 	unsigned int	atime_quantum;
 	signed short	slot;
 	unsigned int	localalloc_opt;
+	unsigned int	resv_level;
 	char		cluster_stack[OCFS2_STACK_LABEL_LEN + 1];
 };
 

@@ -173,6 +174,7 @@ enum {
 	Opt_noacl,
 	Opt_usrquota,
 	Opt_grpquota,
+	Opt_resv_level,
 	Opt_err,
 };
 

@@ -199,6 +201,7 @@ static const match_table_t tokens = {
 	{Opt_noacl, "noacl"},
 	{Opt_usrquota, "usrquota"},
 	{Opt_grpquota, "grpquota"},
+	{Opt_resv_level, "resv_level=%u"},
 	{Opt_err, NULL}
 };
 

@@ -1036,6 +1039,7 @@ static int ocfs2_fill_super(struct super_block *sb, void *data, int silent)
 		     "filesystem does not have the feature enabled.\n");
 		goto read_super_error;
 	}
+	osb->osb_resv_level = parsed_options.resv_level;
 
 	status = ocfs2_verify_userspace_stack(osb, &parsed_options);
 	if (status)

@@ -1262,6 +1266,7 @@ static int ocfs2_parse_options(struct super_block *sb,
 	mopt->slot = OCFS2_INVALID_SLOT;
 	mopt->localalloc_opt = OCFS2_DEFAULT_LOCAL_ALLOC_SIZE;
 	mopt->cluster_stack[0] = '\0';
+	mopt->resv_level = OCFS2_DEFAULT_RESV_LEVEL;
 
 	if (!options) {
 		status = 1;

@@ -1426,6 +1431,18 @@ static int ocfs2_parse_options(struct super_block *sb,
 			printk(KERN_INFO "ocfs2 (no)acl options not supported\n");
 			break;
 #endif
+		case Opt_resv_level:
+			if (is_remount)
+				break;
+			if (match_int(&args[0], &option)) {
+				status = 0;
+				goto bail;
+			}
+			if (option >= OCFS2_MIN_RESV_LEVEL &&
+			    option < OCFS2_MAX_RESV_LEVEL)
+				mopt->resv_level = option;
+			break;
+
 		default:
 			mlog(ML_ERROR,
 			     "Unrecognized mount option \"%s\" "

@@ -1509,6 +1526,9 @@ static int ocfs2_show_options(struct seq_file *s, struct vfsmount *mnt)
 		seq_printf(s, ",noacl");
 #endif
 
+	if (osb->osb_resv_level != OCFS2_DEFAULT_RESV_LEVEL)
+		seq_printf(s, ",resv_level=%d", osb->osb_resv_level);
+
 	return 0;
 }
 

@@ -2037,6 +2057,12 @@ static int ocfs2_initialize_super(struct super_block *sb,
 
 	init_waitqueue_head(&osb->osb_mount_event);
 
+	status = ocfs2_resmap_init(osb, &osb->osb_la_resmap);
+	if (status) {
+		mlog_errno(status);
+		goto bail;
+	}
+
 	osb->vol_label = kmalloc(OCFS2_MAX_VOL_LABEL_LEN, GFP_KERNEL);
 	if (!osb->vol_label) {
 		mlog(ML_ERROR, "unable to alloc vol label\n");
Signature · 19 lines
1.5.6


From 17d6eb73180541c0e73251d838d321f9a44d3492 Mon Sep 17 00:00:00 2001
From: Mark Fasheh <[email protected]>
Date: Mon, 7 Dec 2009 13:15:40 -0800
Subject: [RFC 2/3] ocfs2: use allocation reservations during file write

Add a per-inode reservations structure and pass it through to the
reservations code.

Signed-off-by: Mark Fasheh <[email protected]>
---
 fs/ocfs2/aops.c  |    2 ++
 fs/ocfs2/file.c  |   19 +++++++++++++++++++
 fs/ocfs2/inode.c |    4 ++++
 fs/ocfs2/inode.h |    2 ++
 fs/ocfs2/super.c |    2 ++
 5 files changed, 29 insertions(+), 0 deletions(-)
fs/ocfs2/aops.c +2 -0
index deb2b13..03bae5c 100644
@@ -1760,6 +1760,8 @@ int ocfs2_write_begin_nolock(struct address_space *mapping,
 			goto out;
 		}
 
+		data_ac->ac_resv = &OCFS2_I(inode)->ip_la_data_resv;
+
 		credits = ocfs2_calc_extend_credits(inode->i_sb,
 						    &di->id2.i_list,
 						    clusters_to_alloc);
fs/ocfs2/file.c +19 -0
index de059f4..6a6b7f7 100644
@@ -144,6 +144,7 @@ leave:
 static int ocfs2_file_release(struct inode *inode, struct file *file)
 {
 	struct ocfs2_inode_info *oi = OCFS2_I(inode);
+	struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
 
 	mlog_entry("(0x%p, 0x%p, '%.*s')\n", inode, file,
 		       file->f_path.dentry->d_name.len,

@@ -154,6 +155,21 @@ static int ocfs2_file_release(struct inode *inode, struct file *file)
 		oi->ip_flags &= ~OCFS2_INODE_OPEN_DIRECT;
 	spin_unlock(&oi->ip_lock);
 
+#if 0
+	/*
+	 * Disable this for now. Keeping the reservation around a bit
+	 * longer gives an improvement for workloads which rapidly do
+	 * open()/write()/close() against a file.
+	 */
+	if ((file->f_mode & FMODE_WRITE) &&
+	    (atomic_read(&inode->i_writecount) == 1)) {
+		down_write(&oi->ip_alloc_sem);
+		ocfs2_resv_discard(&osb->osb_la_resmap,
+				   &oi->ip_la_data_resv);
+		up_write(&oi->ip_alloc_sem);
+	}
+#endif
+
 	ocfs2_free_file_private(inode, file);
 
 	mlog_exit(0);

@@ -485,6 +501,9 @@ static int ocfs2_truncate_file(struct inode *inode,
 
 	down_write(&OCFS2_I(inode)->ip_alloc_sem);
 
+	ocfs2_resv_discard(&osb->osb_la_resmap,
+			   &OCFS2_I(inode)->ip_la_data_resv);
+
 	/*
 	 * The inode lock forced other nodes to sync and drop their
 	 * pages, which (correctly) happens even if we have a truncate
fs/ocfs2/inode.c +4 -0
index 0297fb8..98ec32a 100644
@@ -1097,6 +1097,10 @@ void ocfs2_clear_inode(struct inode *inode)
 	ocfs2_mark_lockres_freeing(&oi->ip_inode_lockres);
 	ocfs2_mark_lockres_freeing(&oi->ip_open_lockres);
 
+	ocfs2_resv_discard(&OCFS2_SB(inode->i_sb)->osb_la_resmap,
+			   &oi->ip_la_data_resv);
+	ocfs2_resv_init_once(&oi->ip_la_data_resv);
+
 	/* We very well may get a clear_inode before all an inodes
 	 * metadata has hit disk. Of course, we can't drop any cluster
 	 * locks until the journal has finished with it. The only
fs/ocfs2/inode.h +2 -0
index ba4fe07..e45edca 100644
@@ -70,6 +70,8 @@ struct ocfs2_inode_info
 	/* Only valid if the inode is the dir. */
 	u32				ip_last_used_slot;
 	u64				ip_last_used_group;
+
+	struct ocfs2_alloc_reservation	ip_la_data_resv;
 };
 
 /*
fs/ocfs2/super.c +2 -0
index 62b46a2..dcffe70 100644
@@ -1703,6 +1703,8 @@ static void ocfs2_inode_init_once(void *data)
 	oi->ip_blkno = 0ULL;
 	oi->ip_clusters = 0;
 
+	ocfs2_resv_init_once(&oi->ip_la_data_resv);
+
 	ocfs2_lock_res_init_once(&oi->ip_rw_lockres);
 	ocfs2_lock_res_init_once(&oi->ip_inode_lockres);
 	ocfs2_lock_res_init_once(&oi->ip_open_lockres);
Signature · 15 lines
1.5.6


From ca92a0f7c1c78fff932d766cc391a4ef0b8f66c4 Mon Sep 17 00:00:00 2001
From: Mark Fasheh <[email protected]>
Date: Mon, 7 Dec 2009 13:16:07 -0800
Subject: [RFC 3/3] ocfs2: use allocation reservations for directory data

Use the reservations system for unindexed dir tree allocations. We don't
bother with the indexed tree as reads from it are mostly random anyway.

Signed-off-by: Mark Fasheh <[email protected]>
---
 fs/ocfs2/dir.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/ocfs2/dir.c +2 -0
index 28c3ec2..32b240c 100644
@@ -2993,6 +2993,7 @@ static int ocfs2_expand_inline_dir(struct inode *dir, struct buffer_head *di_bh,
 	 * if we only get one now, that's enough to continue. The rest
 	 * will be claimed after the conversion to extents.
 	 */
+	data_ac->ac_resv = &oi->ip_la_data_resv;
 	ret = ocfs2_claim_clusters(osb, handle, data_ac, 1, &bit_off, &len);
 	if (ret) {
 		mlog_errno(ret);

@@ -3371,6 +3372,7 @@ static int ocfs2_extend_dir(struct ocfs2_super *osb,
 				mlog_errno(status);
 			goto bail;
 		}
+		data_ac->ac_resv = &OCFS2_I(dir)->ip_la_data_resv;
 
 		credits = ocfs2_calc_extend_credits(sb, el, 1);
 	} else {
Signature · 1 line
1.5.6
author
#2 lore raw

RFC 1/3 ocfs2: allocation reservations

In reply to Mark Fasheh

This patch improves Ocfs2 allocation policy by allowing an inode to reserve a portion of the local alloc bitmap for itself. Allocation windows are advisory in that they won't block use of that portion of the bitmap. This makes dealing with corner cases much easier - we can always fall back to previous policy.

Reservation windows are represented internally by a red-black tree. Within that tree, each node represents the reservation window of one inode. When new data is written, we try to allocate from the window first. If that allocation fails, we fall back to our old heuristics and a new window is computed from the results. Allocation windows will also be extended if allocation from them succeeds.

Signed-off-by Mark Fasheh <[email protected]>
---
 Documentation/filesystems/ocfs2.txt |    3 +
 fs/ocfs2/Makefile                   |    1 +
 fs/ocfs2/cluster/masklog.c          |    1 +
 fs/ocfs2/cluster/masklog.h          |    1 +
 fs/ocfs2/localalloc.c               |   39 ++-
 fs/ocfs2/ocfs2.h                    |    5 +
 fs/ocfs2/reservations.c             |  624 +++++++++++++++++++++++++++++++++++
 fs/ocfs2/reservations.h             |  149 +++++++++
 fs/ocfs2/suballoc.h                 |    2 +
 fs/ocfs2/super.c                    |   26 ++
 10 files changed, 845 insertions(+), 6 deletions(-)
 create mode 100644 fs/ocfs2/reservations.c
 create mode 100644 fs/ocfs2/reservations.h
Documentation/filesystems/ocfs2.txt +3 -0
index c58b9f5..0e94600 100644
@@ -80,3 +80,6 @@ user_xattr	(*)	Enables Extended User Attributes.
 nouser_xattr		Disables Extended User Attributes.
 acl			Enables POSIX Access Control Lists support.
 noacl		(*)	Disables POSIX Access Control Lists support.
+resv_level=3	(*)	Set how agressive allocation reservations will be.
+			Valid values are between 0 (reservations off) to 6
+			(maximum space for reservations).
fs/ocfs2/Makefile +1 -0
index 31f25ce..c8eee00 100644
@@ -29,6 +29,7 @@ ocfs2-objs := \
 	mmap.o 			\
 	namei.o 		\
 	refcounttree.o		\
+	reservations.o		\
 	resize.o		\
 	slot_map.o 		\
 	suballoc.o 		\
fs/ocfs2/cluster/masklog.c +1 -0
index 1cd2934..a56147a 100644
@@ -115,6 +115,7 @@ static struct mlog_attribute mlog_attrs[MLOG_MAX_BITS] = {
 	define_mask(ERROR),
 	define_mask(NOTICE),
 	define_mask(KTHREAD),
+	define_mask(RESERVATIONS),
 };
 
 static struct attribute *mlog_attr_ptrs[MLOG_MAX_BITS] = {NULL, };
fs/ocfs2/cluster/masklog.h +1 -0
index 9b4d117..af1a610 100644
@@ -118,6 +118,7 @@
 #define ML_ERROR	0x0000000100000000ULL /* sent to KERN_ERR */
 #define ML_NOTICE	0x0000000200000000ULL /* setn to KERN_NOTICE */
 #define ML_KTHREAD	0x0000000400000000ULL /* kernel thread activity */
+#define	ML_RESERVATIONS	0x0000000800000000ULL /* ocfs2 alloc reservations */
 
 #define MLOG_INITIAL_AND_MASK (ML_ERROR|ML_NOTICE)
 #define MLOG_INITIAL_NOT_MASK (ML_ENTRY|ML_EXIT)
fs/ocfs2/localalloc.c +33 -6
index ac10f83..2042680 100644
@@ -52,7 +52,8 @@ static u32 ocfs2_local_alloc_count_bits(struct ocfs2_dinode *alloc);
 
 static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
 					     struct ocfs2_dinode *alloc,
-					     u32 numbits);
+					     u32 numbits,
+					     struct ocfs2_alloc_reservation *resv);
 
 static void ocfs2_clear_local_alloc(struct ocfs2_dinode *alloc);
 

@@ -262,6 +263,8 @@ void ocfs2_shutdown_local_alloc(struct ocfs2_super *osb)
 
 	osb->local_alloc_state = OCFS2_LA_DISABLED;
 
+	ocfs2_resmap_uninit(&osb->osb_la_resmap);
+
 	main_bm_inode = ocfs2_get_system_file_inode(osb,
 						    GLOBAL_BITMAP_SYSTEM_INODE,
 						    OCFS2_INVALID_SLOT);

@@ -498,7 +501,7 @@ static int ocfs2_local_alloc_in_range(struct inode *inode,
 	alloc = (struct ocfs2_dinode *) osb->local_alloc_bh->b_data;
 	la = OCFS2_LOCAL_ALLOC(alloc);
 
-	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted);
+	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted, NULL);
 	if (start == -1) {
 		mlog_errno(-ENOSPC);
 		return 0;

@@ -664,7 +667,8 @@ int ocfs2_claim_local_alloc_bits(struct ocfs2_super *osb,
 	alloc = (struct ocfs2_dinode *) osb->local_alloc_bh->b_data;
 	la = OCFS2_LOCAL_ALLOC(alloc);
 
-	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted);
+	start = ocfs2_local_alloc_find_clear_bits(osb, alloc, bits_wanted,
+						  ac->ac_resv);
 	if (start == -1) {
 		/* TODO: Shouldn't we just BUG here? */
 		status = -ENOSPC;

@@ -687,6 +691,9 @@ int ocfs2_claim_local_alloc_bits(struct ocfs2_super *osb,
 		goto bail;
 	}
 
+	ocfs2_resmap_claimed_bits(&osb->osb_la_resmap, ac->ac_resv, start,
+				  bits_wanted);
+
 	while(bits_wanted--)
 		ocfs2_set_bit(start++, bitmap);
 

@@ -722,11 +729,13 @@ static u32 ocfs2_local_alloc_count_bits(struct ocfs2_dinode *alloc)
 }
 
 static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
-					     struct ocfs2_dinode *alloc,
-					     u32 numbits)
+				     struct ocfs2_dinode *alloc,
+				     u32 numbits,
+				     struct ocfs2_alloc_reservation *resv)
 {
 	int numfound, bitoff, left, startoff, lastzero;
 	void *bitmap = NULL;
+	struct ocfs2_reservation_map *resmap = &osb->osb_la_resmap;
 
 	mlog_entry("(numbits wanted = %u)\n", numbits);
 

@@ -738,6 +747,20 @@ static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
 
 	bitmap = OCFS2_LOCAL_ALLOC(alloc)->la_bitmap;
 
+	/*
+	 * Ask the reservations code first whether this request can be
+	 * easily fulfilled. No errors here are fatal - if we didn't
+	 * find the number of bits needed, we'll just take the slow
+	 * path.
+	 */
+	if (ocfs2_resmap_resv_bits(resmap, resv, bitmap, &bitoff, &numfound)
+	    == 0) {
+		if (numfound >= numbits) {
+			numfound = numbits;
+			goto bail;
+		}
+	}
+
 	numfound = bitoff = startoff = 0;
 	lastzero = -1;
 	left = le32_to_cpu(alloc->id1.bitmap1.i_total);

@@ -772,8 +795,10 @@ static int ocfs2_local_alloc_find_clear_bits(struct ocfs2_super *osb,
 
 	if (numfound == numbits)
 		bitoff = startoff - numfound;
-	else
+	else {
+		numfound = 0;
 		bitoff = -1;
+	}
 
 bail:
 	mlog_exit(bitoff);

@@ -1096,6 +1121,8 @@ retry_enospc:
 	memset(OCFS2_LOCAL_ALLOC(alloc)->la_bitmap, 0,
 	       le16_to_cpu(la->la_size));
 
+	ocfs2_resmap_restart(&osb->osb_la_resmap, cluster_count);
+
 	mlog(0, "New window allocated:\n");
 	mlog(0, "window la_bm_off = %u\n",
 	     OCFS2_LOCAL_ALLOC(alloc)->la_bm_off);
fs/ocfs2/ocfs2.h +5 -0
index d963d86..5a6300f 100644
@@ -46,6 +46,7 @@
 /* For struct ocfs2_blockcheck_stats */
 #include "blockcheck.h"
 
+#include "reservations.h"
 
 /* Caching of metadata buffers */
 

@@ -340,6 +341,10 @@ struct ocfs2_super
 
 	u64 la_last_gd;
 
+	struct ocfs2_reservation_map	osb_la_resmap;
+
+	unsigned int	osb_resv_level;
+
 	/* Next three fields are for local node slot recovery during
 	 * mount. */
 	int dirty;
fs/ocfs2/reservations.c new file +624 -0

624 lines not on this page · raw message

fs/ocfs2/reservations.h new file +149 -0

149 lines not on this page · raw message

fs/ocfs2/suballoc.h +2 -0
index 8c9a78a..5eb7753 100644
@@ -54,6 +54,8 @@ struct ocfs2_alloc_context {
 	u64    ac_last_group;
 	u64    ac_max_block;  /* Highest block number to allocate. 0 is
 				 is the same as ~0 - unlimited */
+
+	struct ocfs2_alloc_reservation	*ac_resv;
 };
 
 void ocfs2_free_alloc_context(struct ocfs2_alloc_context *ac);
fs/ocfs2/super.c +26 -0
index 14f47d2..62b46a2 100644
@@ -94,6 +94,7 @@ struct mount_options
 	unsigned int	atime_quantum;
 	signed short	slot;
 	unsigned int	localalloc_opt;
+	unsigned int	resv_level;
 	char		cluster_stack[OCFS2_STACK_LABEL_LEN + 1];
 };
 

@@ -173,6 +174,7 @@ enum {
 	Opt_noacl,
 	Opt_usrquota,
 	Opt_grpquota,
+	Opt_resv_level,
 	Opt_err,
 };
 

@@ -199,6 +201,7 @@ static const match_table_t tokens = {
 	{Opt_noacl, "noacl"},
 	{Opt_usrquota, "usrquota"},
 	{Opt_grpquota, "grpquota"},
+	{Opt_resv_level, "resv_level=%u"},
 	{Opt_err, NULL}
 };
 

@@ -1036,6 +1039,7 @@ static int ocfs2_fill_super(struct super_block *sb, void *data, int silent)
 		     "filesystem does not have the feature enabled.\n");
 		goto read_super_error;
 	}
+	osb->osb_resv_level = parsed_options.resv_level;
 
 	status = ocfs2_verify_userspace_stack(osb, &parsed_options);
 	if (status)

@@ -1262,6 +1266,7 @@ static int ocfs2_parse_options(struct super_block *sb,
 	mopt->slot = OCFS2_INVALID_SLOT;
 	mopt->localalloc_opt = OCFS2_DEFAULT_LOCAL_ALLOC_SIZE;
 	mopt->cluster_stack[0] = '\0';
+	mopt->resv_level = OCFS2_DEFAULT_RESV_LEVEL;
 
 	if (!options) {
 		status = 1;

@@ -1426,6 +1431,18 @@ static int ocfs2_parse_options(struct super_block *sb,
 			printk(KERN_INFO "ocfs2 (no)acl options not supported\n");
 			break;
 #endif
+		case Opt_resv_level:
+			if (is_remount)
+				break;
+			if (match_int(&args[0], &option)) {
+				status = 0;
+				goto bail;
+			}
+			if (option >= OCFS2_MIN_RESV_LEVEL &&
+			    option < OCFS2_MAX_RESV_LEVEL)
+				mopt->resv_level = option;
+			break;
+
 		default:
 			mlog(ML_ERROR,
 			     "Unrecognized mount option \"%s\" "

@@ -1509,6 +1526,9 @@ static int ocfs2_show_options(struct seq_file *s, struct vfsmount *mnt)
 		seq_printf(s, ",noacl");
 #endif
 
+	if (osb->osb_resv_level != OCFS2_DEFAULT_RESV_LEVEL)
+		seq_printf(s, ",resv_level=%d", osb->osb_resv_level);
+
 	return 0;
 }
 

@@ -2037,6 +2057,12 @@ static int ocfs2_initialize_super(struct super_block *sb,
 
 	init_waitqueue_head(&osb->osb_mount_event);
 
+	status = ocfs2_resmap_init(osb, &osb->osb_la_resmap);
+	if (status) {
+		mlog_errno(status);
+		goto bail;
+	}
+
 	osb->vol_label = kmalloc(OCFS2_MAX_VOL_LABEL_LEN, GFP_KERNEL);
 	if (!osb->vol_label) {
 		mlog(ML_ERROR, "unable to alloc vol label\n");
Signature · 1 line
1.5.6
#3 lore raw

RFC 1/3 ocfs2: allocation reservations

Quoted 6 lines · Mark Fasheh
fs/ocfs2/reservations.c new file +0 -0
new file mode 100644
index 0000000..8879e28

fs/ocfs2/reservations.c line 8

Quoted 8 lines
fs/ocfs2/reservations.c new file +8 -0
@@ -0,0 +1,624 @@
+/* -*- mode: c; c-basic-offset: 8; -*-
+ * vim: noexpandtab sw=8 ts=8 sts=0:
+ *
+ * reservations.c
+ *
+ * Allocation reservations implementation
+ *
+ * Some code borrowed from fs/ext3/balloc.c

Does that file have copyright info. Probably should include
that here.

Quoted 7 lines
fs/ocfs2/reservations.c new file +7 -0
+ *
+ * Copyright (C) 2009 Novell.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.

fs/ocfs2/reservations.c line 25

Quoted 10 lines
fs/ocfs2/reservations.c new file +10 -0
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.

Drop this last paragraph. It's no longer accurate, and GNU says
drop it rather than correct it.

Quoted 4 lines
fs/ocfs2/reservations.c new file +4 -0
+static void ocfs2_resmap_clear_all_resv(struct ocfs2_reservation_map *resmap)
+{
+	struct rb_node *node;
+	struct ocfs2_alloc_reservation *resv;

fs/ocfs2/reservations.c line 201

Quoted 10 lines
fs/ocfs2/reservations.c new file +10 -0
+
+	assert_spin_locked(&resv_lock);
+
+	while ((node = rb_last(&resmap->m_reservations)) != NULL) {
+		resv = rb_entry(node, struct ocfs2_alloc_reservation, r_node);
+
+		__ocfs2_resv_trunc(resv);
+		rb_erase(&resv->r_node, &resmap->m_reservations);
+	}
+}

Ugh, this always walks to the end of the tree on each iteration
of the while loop. That's O(nlogn). But it looks like everyone does this. Joy.

fs/ocfs2/reservations.c line 343

Quoted 4 lines
fs/ocfs2/reservations.c new file +4 -0
+static struct ocfs2_alloc_reservation *
+ocfs2_find_resv(struct ocfs2_reservation_map *resmap, unsigned int goal)
+static void ocfs2_resv_find_window(struct ocfs2_reservation_map *resmap,
+				   struct ocfs2_alloc_reservation *resv)

At first this confused me, but then I figured out you're asking
"where can I stick my reservation?" This is why the reservation object doesn't need a backref to the inode.

Joel

Signature · 9 lines
Life's Little Instruction Book #407

	"Every once in a while, take the scenic route."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
author
#4 lore raw

RFC 2/3 ocfs2: use allocation reservations during file write

In reply to Mark Fasheh

Add a per-inode reservations structure and pass it through to the reservations code.

Signed-off-by Mark Fasheh <[email protected]>
---
 fs/ocfs2/aops.c  |    2 ++
 fs/ocfs2/file.c  |   19 +++++++++++++++++++
 fs/ocfs2/inode.c |    4 ++++
 fs/ocfs2/inode.h |    2 ++
 fs/ocfs2/super.c |    2 ++
 5 files changed, 29 insertions(+), 0 deletions(-)
fs/ocfs2/aops.c +2 -0
index deb2b13..03bae5c 100644
@@ -1760,6 +1760,8 @@ int ocfs2_write_begin_nolock(struct address_space *mapping,
 			goto out;
 		}
 
+		data_ac->ac_resv = &OCFS2_I(inode)->ip_la_data_resv;
+
 		credits = ocfs2_calc_extend_credits(inode->i_sb,
 						    &di->id2.i_list,
 						    clusters_to_alloc);
fs/ocfs2/file.c +19 -0
index de059f4..6a6b7f7 100644
@@ -144,6 +144,7 @@ leave:
 static int ocfs2_file_release(struct inode *inode, struct file *file)
 {
 	struct ocfs2_inode_info *oi = OCFS2_I(inode);
+	struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
 
 	mlog_entry("(0x%p, 0x%p, '%.*s')\n", inode, file,
 		       file->f_path.dentry->d_name.len,

@@ -154,6 +155,21 @@ static int ocfs2_file_release(struct inode *inode, struct file *file)
 		oi->ip_flags &= ~OCFS2_INODE_OPEN_DIRECT;
 	spin_unlock(&oi->ip_lock);
 
+#if 0
+	/*
+	 * Disable this for now. Keeping the reservation around a bit
+	 * longer gives an improvement for workloads which rapidly do
+	 * open()/write()/close() against a file.
+	 */
+	if ((file->f_mode & FMODE_WRITE) &&
+	    (atomic_read(&inode->i_writecount) == 1)) {
+		down_write(&oi->ip_alloc_sem);
+		ocfs2_resv_discard(&osb->osb_la_resmap,
+				   &oi->ip_la_data_resv);
+		up_write(&oi->ip_alloc_sem);
+	}
+#endif
+
 	ocfs2_free_file_private(inode, file);
 
 	mlog_exit(0);

@@ -485,6 +501,9 @@ static int ocfs2_truncate_file(struct inode *inode,
 
 	down_write(&OCFS2_I(inode)->ip_alloc_sem);
 
+	ocfs2_resv_discard(&osb->osb_la_resmap,
+			   &OCFS2_I(inode)->ip_la_data_resv);
+
 	/*
 	 * The inode lock forced other nodes to sync and drop their
 	 * pages, which (correctly) happens even if we have a truncate
fs/ocfs2/inode.c +4 -0
index 0297fb8..98ec32a 100644
@@ -1097,6 +1097,10 @@ void ocfs2_clear_inode(struct inode *inode)
 	ocfs2_mark_lockres_freeing(&oi->ip_inode_lockres);
 	ocfs2_mark_lockres_freeing(&oi->ip_open_lockres);
 
+	ocfs2_resv_discard(&OCFS2_SB(inode->i_sb)->osb_la_resmap,
+			   &oi->ip_la_data_resv);
+	ocfs2_resv_init_once(&oi->ip_la_data_resv);
+
 	/* We very well may get a clear_inode before all an inodes
 	 * metadata has hit disk. Of course, we can't drop any cluster
 	 * locks until the journal has finished with it. The only
fs/ocfs2/inode.h +2 -0
index ba4fe07..e45edca 100644
@@ -70,6 +70,8 @@ struct ocfs2_inode_info
 	/* Only valid if the inode is the dir. */
 	u32				ip_last_used_slot;
 	u64				ip_last_used_group;
+
+	struct ocfs2_alloc_reservation	ip_la_data_resv;
 };
 
 /*
fs/ocfs2/super.c +2 -0
index 62b46a2..dcffe70 100644
@@ -1703,6 +1703,8 @@ static void ocfs2_inode_init_once(void *data)
 	oi->ip_blkno = 0ULL;
 	oi->ip_clusters = 0;
 
+	ocfs2_resv_init_once(&oi->ip_la_data_resv);
+
 	ocfs2_lock_res_init_once(&oi->ip_rw_lockres);
 	ocfs2_lock_res_init_once(&oi->ip_inode_lockres);
 	ocfs2_lock_res_init_once(&oi->ip_open_lockres);
Signature · 1 line
1.5.6
author
#5 lore raw

RFC 3/3 ocfs2: use allocation reservations for directory data

In reply to Mark Fasheh

Use the reservations system for unindexed dir tree allocations. We don't bother with the indexed tree as reads from it are mostly random anyway.

Signed-off-by Mark Fasheh <[email protected]>
---
 fs/ocfs2/dir.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/ocfs2/dir.c +2 -0
index 28c3ec2..32b240c 100644
@@ -2993,6 +2993,7 @@ static int ocfs2_expand_inline_dir(struct inode *dir, struct buffer_head *di_bh,
 	 * if we only get one now, that's enough to continue. The rest
 	 * will be claimed after the conversion to extents.
 	 */
+	data_ac->ac_resv = &oi->ip_la_data_resv;
 	ret = ocfs2_claim_clusters(osb, handle, data_ac, 1, &bit_off, &len);
 	if (ret) {
 		mlog_errno(ret);

@@ -3371,6 +3372,7 @@ static int ocfs2_extend_dir(struct ocfs2_super *osb,
 				mlog_errno(status);
 			goto bail;
 		}
+		data_ac->ac_resv = &OCFS2_I(dir)->ip_la_data_resv;
 
 		credits = ocfs2_calc_extend_credits(sb, el, 1);
 	} else {
Signature · 1 line
1.5.6
#6 lore raw

cover Ocfs2 allocation reservations

In reply to Mark Fasheh
Quoted 7 lines · Mark Fasheh

Hi, the following patches implement allocation reservations for Ocfs2. Right now I'd consider them to be beta quality - simple operations work but the code likely won't survive a kernel build. As it is implemented, the code only takes reservations on the local alloc bitmap. I've divorced the structures from any bitmap type however, so we can extend this to the chain allocators with only a small amount of code. Also, there's nothing in reservations.c which depends on a struct inode.

It looks nice and simple. As I read it, when you get some space
from the localalloc, you try to reserve the next space. Simple chaining of requests. It's efficacy obviously depends on how much you reserve for the next cycle.

Quoted 5 lines

This version of the reservations code shows the following improvements:

Inode: 66007 % fragmented: 50.00 clusters: 32 extents: 16
Inode: 66009 % fragmented: 68.75 clusters: 32 extents: 22
Inode: 66008 % fragmented: 43.75 clusters: 32 extents: 14

This is probably plenty good. Here's hoping the knob allows us
to tune it. We can probably run with different tunings to see where we come out best.
Have you seen any deleterious effects from very aggressive
reservations?

Joel

Signature · 8 lines
"Time is an illusion, lunchtime doubly so."
        -Douglas Adams

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127