Ext4 devel interlock meeting minutes (April 23, 2007)

Ext4 Developer Interlock Call: 04/23/2007 Meeting Minutes

Attendees: Mingming Cao, Dave Kleikamp, Avantika Mathur, Ted Ts'o, Suparna Bhattacharya, Jean-Pierre Dion, Jean Noel Cordenner, Valérie Clément, Jose Santos

Minutes can be accessed at: http://ext4.wiki.kernel.org/index.php/Ext4_Developer%27s_Conference_Call

- Mingming proposed moving back to 8am PST meeting time, since the 6am time is inconvenient for a few people. This discussion will be continued through email, to find a time which works for everyone.

- Next week's meeting will be canceled, unless there is anyone who would like to request a meeting.

PATCH STATUS

git-tree
- Mingming will be updating the git tree with extents-fix patches from Alex, i_flags patch from Honza, i_extra_isize patch from Kalpak.

Uninitialized Block Groups:
- The patch sent out by Andreas is against 2.6.16 and ext3. Need to port this to current ext4, test and then add to git-tree. Avantika will ask Andreas if he needs help with this.

JBD statistics:
- There is a patch to export JDB statistics to /proc. In order to get this patch to mainline, there needs to be discussion about the correct place for the statistics; /proc or perhaps debugfs.

e2fsprogs:
- Ted will post the current e2fsprogs patches in progress. Ted has been working with these patches and making changes.
- Main work areas for making e2fsprogs compatible with extents and 64-bit.
- block iterator: make a block iterator work with both extent and
non-extent code. Code that is oblivious to extents will still work with the block iterator. This has been written by Andreas Dilger.
- extents: in order to preserve ABI compatibility, support for a new
interface for extents which uses 64-bit logical and physical block numbers. The block iterator then translate from on-disk to in-memory format. This will allow for possible future increases of physical and logical block sizes in extents, without breaking ABI.
- bitmaps in e2fsprogs: this will be discussed in more detail at the
next meeting, after people have a chance to read related email.

preallocation:
- fallocate syscall interface: the current plan, based on discussions on the mailing list, is to create a separate wrapper for s390 in glibc. Using regular parameter ordering for all other architectures, but a different order on s390. Jakub Jelinek has said that the changes in glibc can be made pretty easily.
- The preallocation patches in the ext4 git-tree are outdated, using the ioctl interface. Once Amit re-posts the patches with the syscall interface, they will be updated in the git-tree as well.
- Mingming mentioned the need to flush preallocation metadata changes to disk if file size or file content is being tested. Discussed doing an fsync at Bmap time.

TESTING
- extents testing
- Discussed methods for testing extents on highly fragmented
filesystems.
- Jose will look into possible tests, including perhaps using the
'aged' option in FFSB
- Ted suggested creating a mountoption that creates a bad block
allocator which it jumps to a new block group every 8 blocks. This would force a very large number of extents, and may be a good test for extents.

- large filesystem
- We would like to perform more testing on large (>16TB) filesystems
- currently hardware limitations are preventing this testing. We
have tested 10TB raid dists, and 16TB loopback devices. Avantika will look into creating very large sparse devices for testing.

- Large file deletion
- Valerie had recently tested large file deletion on ext3/4, but did
not see the expected performance gain with ext4 due to compact metadata when using extents.
- Valerie will try re-running the test. Jose will also be looking
into this test.

there is AGGRESSIVE_TEST define which limits number of entries in index/leaf.

any details?

thanks, Alex

Ok, I found my mistake. There was a typo in my test script and the pagecache was not flushed between the file creation and the deletion.

Here are the results I obtain with a 2.6.17-rc7 kernel to delete a 100GB file:

ext3 : real  2m35.048s    user  0m0.000s     sys  0m6.424s
ext4 : real  0m11.160s    user  0m0.000s     sys  0m5.532s
xfs :  real  0m0.377s     user  0m0.004s     sys  0m0.004s

The performance gain with ext4 is much larger when running a good test... Sorry the wrong information,

Valérie

would be very interesting to know how much IO was done to remove the file and actual fragmentation in all the cases.

thanks, Alex

I've been hacking up some ext3@16T testing scripts to use sparse devicemapper devices which make use of snapshots... loopback files don't work for testing, at least not hosted on ext[234], because we still can't do these large file offsets.

(Documentation/device-mapper/zero.txt in the kernel tree describes these sparse dm devices)

Testing the whole range as a sparse snapshot can be slow, since devicemapper has to do all the exception handling etc, and I think essentially creates a fragmented block device.

I've been playing with something like this:

# 90% of the real device size is used for a "real" 1:1 mapping # The other 10% is sparsely mapped out to add up to totalsize.

# i.e. -

# [large sparse-ish device]
#
# +----------------------~ ~-----------------------------------------+
# | sparse | real |
# +----------------------~ ~-----------------------------------------+ #
# |<------------ SPARSE_SIZE ---------------->|<----- REAL_SIZE ----->|

# is mapped on top of:

# [real block device]

#                      +----------------------------+
#                      | sp |       real            |
#                      +----------------------------+

and then marking the sparse range as full (maybe via lazy_bg, or other methods). You could then also put a dm-error target under the "full" sections so that any IO that may stray there will fail.

This way you can direct the real IO to the 1:1 mapping portion of the large dm device, and shouldn't get the snapshot slowdowns.

Anyway, just something I've been playing with...

-eric

Ok, I will do it.

Valérie

What i am doing for creating a large number of extents is

dd if=/dev/zero of=myfile count=10
seek=20
while [ 1 ]; do dd if=/dev/zero of=myfile count=10 seek=$seek; seek=expr $seek + 20; done

-aneesh

with AGGRESSIVE_TEST defined in include/linux/ext4_fs_extents.h you may get much more extents and index blocks.

I had written a simple tool "bitmap_manip" with which you can actually manipulate the number of free chunks and their sizes in a filesystem. It uses libext2fs to set the bits in block bitmaps thereby leaving the desired free extents. I had written it to test the allocators performance.

It can be used as:
./bitmap_manip /dev/sda9 1MA 4 16K 1 12K 3 8K 4 4K 6

This will leave only 1 16K chunk, 3 12K chunks, .... free in the filesystem. "1MA" 4 will get us 4 1Mb free ALIGNED chunks.

It isn't very beautiful code since it was only used for testing but maybe it can help.

Thanks,
Kalpak.

/* Manipulate block bitmap directly for mballoc testing */

/* USAGE:
* ./bitmap_manip /dev/volmballoc/test 16K 1 12K 3 8K 4 4K 6
* This will leave 1 16K chunk, 3 12K chunks, .... in the filesystem specified.
* Ideally give the inputs in ascending order.
* 1MA 4 will get us 4 1Mb ALIGNED chunks.
*/

#include <stdio.h>
#include <ext2fs/ext2fs.h>
#include <ext2fs/ext2_types.h>
#include <fcntl.h>
#include <stdlib.h>

#define ONE_MB (1024 * 1024)
#define ONE_KB 1024

#define SETTING 0
#define FREEING 1

#define NO_ALIGN 0
#define ALIGN 1

struct chunk_arg {
	int chunk_size;
	int num_chunks;
	int align;
};

int main(int argc, char **argv)

{
	ext2_filsys fs;
	ext2fs_block_bitmap *map = NULL;
	int bg_num = 0, retval, arg_num, multiply, chunk_num;
	int i, start_blk, set_bit, test_bit, j;
	struct chunk_arg chunk[50];
	int free_blocks_req = 0, free_blocks_avail, num_of_chunks_req = 0, group;
	char str[10];
	float orig_avail_req, avail_req;
	int set_till_now, free_till_now, num_blks_to_set, num_blks_to_free, phase;
	int  current, align_flag = 0, align = 0, curr = 0;

	if (argc < 2) {
		printf("Please give name of a filesystem. Exiting...\n");
		return -1;
	}
	
	/* Even from user's perspective */
	if(argc & 0x01) {
		printf("This utility cannot have even number of arguments.\n");
		return -1;
	}

	if ((retval = ext2fs_open(argv[1], EXT2_FLAG_RW, 0, 0, unix_io_manager, &fs))) {
		com_err("ext2fs open:", retval, "while opening %s\n", argv[1]);
		return retval;
	}	

	srand(1234567);
	chunk_num = 0;
	for (arg_num = 2; arg_num < argc; arg_num += 2, chunk_num++) {
		strcpy(str, argv[arg_num]);

		/* Check if we have to align */
		if (toupper(str[strlen(str) - 1 ]) == 'A') {
			chunk[chunk_num].align = ALIGN;
			str[strlen(str) - 1] = '\0';	
			align = 1;
		}
		else
			chunk[chunk_num].align = NO_ALIGN;
		if (toupper(str[strlen(str) - 1]) == 'K')
			multiply = ONE_KB;
		else if(toupper(str[strlen(str) - 1]) == 'M') 
			multiply = ONE_MB;

		str[strlen(str) - 1] = '\0';
		chunk[chunk_num].chunk_size = ((strtod(str, NULL)) * multiply)/ (fs->blocksize);
		chunk[chunk_num].num_chunks = strtod(argv[arg_num + 1], NULL); 
			
		free_blocks_req += chunk[chunk_num].chunk_size * chunk[chunk_num].num_chunks;
		num_of_chunks_req += chunk[chunk_num].num_chunks;
	}

	ext2fs_read_block_bitmap(fs);			
	map = &fs->block_map;	

	start_blk = fs->super->s_first_data_block;
	free_blocks_avail = fs->super->s_free_blocks_count;

	orig_avail_req = free_blocks_avail / free_blocks_req;
	current = 0;
	i = start_blk;
	
	num_blks_to_set = (orig_avail_req / 4) * chunk[current].chunk_size;
	num_blks_to_free = chunk[current].chunk_size;
	phase = SETTING;
	do {
		test_bit = i;
		if (!ext2fs_fast_test_block_bitmap(*map, test_bit)) {
			if (phase == SETTING) {				
				if (chunk[current].align == ALIGN && chunk[current].num_chunks > 0) {
					if (align_flag == 0) {
						num_blks_to_set = (i / chunk[current].chunk_size + 1) * 
							chunk[current].chunk_size - i;
						align_flag = 1;
					}
					else if (i % chunk[current].chunk_size == 0) {
						num_blks_to_set = 0;
						phase = FREEING;
					}
				}
				set_bit = i;
		                ext2fs_mark_block_bitmap(*map, set_bit); 
				group = (set_bit - fs->super->s_first_data_block) / fs->super->s_blocks_per_group;
				fs->group_desc[group].bg_free_blocks_count--;
				fs->super->s_free_blocks_count--;		
				num_blks_to_set--;
				if (num_blks_to_set == 0) {
					phase = FREEING;
					align_flag = 0;
				}
			}
			else if (phase == FREEING) { 
				free_blocks_req--;
				num_blks_to_free--;
				if (num_blks_to_free == 0) {
					/* Decide how many blocks to set */
					phase = SETTING;
					num_of_chunks_req--;
					chunk[current].num_chunks--;
					
					/* No more free chunks required*/
					if (num_of_chunks_req == 0) {
						num_blks_to_set = free_blocks_avail;
					}
					else {
						for (j = 0; j < chunk_num; j++) {	
							if (chunk[j].num_chunks > 0) {	
								if (free_blocks_req > chunk[j].num_chunks * 
 									    chunk[j].chunk_size && current == j) {
									continue;
								}
								else {
									current = j;
									break;	
								}
							}
						}							
  					        avail_req = free_blocks_avail / free_blocks_req;
						if (align != 1)
	       						num_blks_to_set = (avail_req / 4) * 
								    chunk[current].chunk_size;
						else 
							num_blks_to_set = 20;
						num_blks_to_free = chunk[current].chunk_size;		
						/* Make sure a free block does not break across block groups */
						curr = i % 32767;
						curr = 32767 * (curr + 1);
						if (i + num_blks_to_set + num_blks_to_free > curr && i < curr)	
							num_blks_to_set += (curr) - (i + num_blks_to_set);
					}
				}
			}												
			free_blocks_avail--;	
		}
                i++;
	}while(i <= (fs->super->s_blocks_count - 1) || free_blocks_avail != 0);

        ext2fs_mark_bb_dirty(fs);
        ext2fs_mark_super_dirty(fs);

	if (i == fs->super->s_blocks_count && free_blocks_req != 0) {
		printf("Block manipulation failed. Sorry.\n");
		return 0;
	}		

	ext2fs_close(fs);	
	return 0;
}