question about ext4 block allocation

I recently hit an issue in my DAX testing where I was unable to get ext4 to give me 2 MiB sized and aligned block allocations in a situation where I thought I should be able to. I'm using a PMEM ramdisk of size 16 GiB, created using the memmap kernel command line parameter.

  # fdisk -l /dev/pmem0
  Disk /dev/pmem0: 16 GiB, 17179869184 bytes, 33554432 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 4096 bytes
  I/O size (minimum/optimal): 4096 bytes / 4096 bytes

The very simple test program I used to reproduce this can be found at the bottom of this mail. Here is the quick function that I used to recreate my filesystem each run:

  # type go_ext4
  go_ext4 is a function
  go_ext4 () 
  { 
      umount /dev/pmem0 2> /dev/null;
      mkfs.ext4 -b 4096 -E stride=512 -F /dev/pmem0;
      mount -o dax /dev/pmem0 ~/dax;
      cd ~/fsync
  }

To be able to easily see whether DAX is able to use PMDs instead of PTEs, you can run with the mmots tree (git://git.cmpxchg.org/linux-mmots.git), tag v4.10-rc4-mmots-2017-01-17-16-32.

Okay, so here's the interesting part. If I create a filesystem and run the test so it creates a file of size 32 MiB or 128 MiB, I get a PMD fault. Here's the corresponding tracepoint output:

test-1429 [008] .... 10573.026699: dax_pmd_fault: dev 259:0 ino 0xc shared WRITE|ALLOW_RETRY|KILLABLE|USER address 0x40280000 vm_start 0x40000000 vm_end 0x40400000 pgoff 0x280 max_pgoff 0x7fff

test-1429 [008] .... 10573.026912: dax_pmd_insert_mapping: dev 259:0 ino 0xc shared write address 0x40280000 length 0x200000 pfn 0x108a00 DEV|MAP radix_entry 0x114000e

test-1429 [008] .... 10573.026917: dax_pmd_fault_done: dev 259:0 ino 0xc shared WRITE|ALLOW_RETRY|KILLABLE|USER address 0x40280000 vm_start 0x40000000 vm_end 0x40400000 pgoff 0x280 max_pgoff 0x7fff NOPAGE

Great. That's what I want. But, if I create the filesystem and use the test to create a file that is 64 MiB in size, the PMD fault fails because the PFN I get from the filesystem isn't 2MiB aligned:

test-1475 [006] .... 11809.982188: dax_pmd_fault: dev 259:0 ino 0xc shared WRITE|ALLOW_RETRY|KILLABLE|USER address 0x40280000 vm_start 0x40000000 vm_end 0x40400000 pgoff 0x280 max_pgoff 0x3fff

test-1475 [006] .... 11809.982398: dax_pmd_insert_mapping_fallback: dev 259:0 ino 0xc shared write address 0x40280000 length 0x200000 pfn 0x108601 DEV|MAP radix_entry 0x0

test-1475 [006] .... 11809.982399: dax_pmd_fault_done: dev 259:0 ino 0xc shared WRITE|ALLOW_RETRY|KILLABLE|USER address 0x40280000 vm_start 0x40000000 vm_end 0x40400000 pgoff 0x280 max_pgoff 0x3fff FALLBACK

The PFN for the block allocation I get from ext4 is 0x108601, which isn't aligned, so we fail the PG_PMD_COLOUR alignment check in dax_iomap_pmd_fault(), and use PTEs instead.

I initially saw this in a test from Xiong:

https://www.mail-archive.com/[email protected]/msg02615.html

and created the attached test to have a simpler reproducer. With Xiong's test, a test on a 128 MiB sized file will have all PMDs, an on a 64 MiB file we'll use all PTEs.

This question is important because eventually we'd like to say to customers "do X and you should get PMDs when you use DAX", but right now I'm not sure what X is. :)

Thanks,
- Ross

--- >8 ---
#define _GNU_SOURCE
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <linux/falloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>

#define GiB(a) ((a)*1024ULL*1024*1024)
#define MiB(a) ((a)*1024ULL*1024)
#define PAGE(a) ((a)*0x1000)

void usage(char *prog)

{
	fprintf(stderr, "usage: %s <size in MiB>\n", prog);
	exit(1);
}

void err_exit(char *op, unsigned long len)

{
	fprintf(stderr, "%s(%s) len %lu\n", op, strerror(errno), len);
	exit(1);
}

int main(int argc, char *argv[])

{
	char *data_array = (char*) GiB(1); /* request a 2MiB aligned address with mmap() */
	unsigned long len;
	int fd;

	if (argc < 2)
		usage(basename(argv[0]));

	len = strtoul(argv[1], NULL, 10);
	if (errno == ERANGE)
		err_exit("strtoul", 0);

	fd = open("/root/dax/data", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR);
	if (fd < 0) {
		perror("fd");
		return 1;
	}

	ftruncate(fd, 0);
	fallocate(fd, 0, 0, MiB(len));

	data_array = mmap(data_array, PAGE(0x400), PROT_READ|PROT_WRITE,
			MAP_SHARED, fd, PAGE(0));
	data_array[PAGE(0x280)] = 142;

	fsync(fd);
	close(fd);
	return 0;
}

Hi Ross,

...

Yeah, it's a bug in ext4 allocator. Requests for 128MB are exactly a group size so we find completely empty group and satisfy the request. Even larger requests will get split into 128MB chunks. 32MB requests are small enough that they go via a special path for power-of-two sized requests. However 64MB allocation request can be satisfied from somewhat filled group (there are sb backup blocks in group 1 in your case) and we screw up when deciding whether to treat such request as power-of-two or not and don't align it at all in the end.

Another problem is that the stride size ends up unused due to another bug in ext4. The second attached patch fixes that issue.

With these two patches applied I get file blocks aligned. That being said the stripe-aligned allocator does a poor job of creating large extents (larger than stripe-width) however that is more difficult to fix.

Honza

Jan Kara <jack-IBi9RG/[email protected]>
SUSE Labs, CR

From: Jan Kara <[email protected]>
Date: Thu, 9 Feb 2017 15:46:46 +0100
Subject: [PATCH 1/2] ext4: Fix stripe-unaligned allocations

When a filesystem is created using:

mkfs.ext4 -b 4096 -E stride=512 <dev>

and we try to allocate 64MB extent, we will end up directly in ext4_mb_complex_scan_group(). This is because the request is detected as power-of-two allocation (so we start in ext4_mb_regular_allocator() with ac_criteria == 0) however the check before ext4_mb_simple_scan_group() refuses the direct buddy scan because the allocation request is too large. Since cr == 0, the check whether we should use ext4_mb_scan_aligned() fails as well and we fall back to ext4_mb_complex_scan_group().

Fix the problem by checking for upper limit on power-of-two requests directly when detecting them.

---
 fs/ext4/mballoc.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

index 7ae43c59bc79..37db7ba0f69e 100644
@@ -2136,8 +2136,10 @@ ext4_mb_regular_allocator(struct ext4_allocation_context *ac)
 	 * We search using buddy data only if the order of the request
 	 * is greater than equal to the sbi_s_mb_order2_reqs
 	 * You can tune it via /sys/fs/ext4/<partition>/mb_order2_req
+	 * We also support searching for power-of-two requests only for
+	 * requests upto maximum buddy size we have constructed.
 	 */
-	if (i >= sbi->s_mb_order2_reqs) {
+	if (i >= sbi->s_mb_order2_reqs && i <= sb->s_blocksize_bits + 2) {
 		/*
 		 * This should tell if fe_len is exactly power of 2
 		 */

@@ -2207,7 +2209,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_context *ac)
 			}
 
 			ac->ac_groups_scanned++;
-			if (cr == 0 && ac->ac_2order < sb->s_blocksize_bits+2)
+			if (cr == 0)
 				ext4_mb_simple_scan_group(ac, &e4b);
 			else if (cr == 1 && sbi->s_stripe &&
 					!(ac->ac_g_ex.fe_len % sbi->s_stripe))

2.10.2

From: Jan Kara <[email protected]>
Date: Thu, 9 Feb 2017 16:13:30 +0100
Subject: [PATCH 2/2] ext4: Do not use stripe_with if it is not set

Avoid using stripe_width for sbi->s_stripe value if it is not actually set. It prevents using the stride for sbi->s_stripe.

---
 fs/ext4/super.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 66845a08a87a..b82cd3b263b4 100644
@@ -2619,9 +2619,9 @@ static unsigned long ext4_get_stripe_size(struct ext4_sb_info *sbi)
 
 	if (sbi->s_stripe && sbi->s_stripe <= sbi->s_blocks_per_group)
 		ret = sbi->s_stripe;
-	else if (stripe_width <= sbi->s_blocks_per_group)
+	else if (stripe_width && stripe_width <= sbi->s_blocks_per_group)
 		ret = stripe_width;
-	else if (stride <= sbi->s_blocks_per_group)
+	else if (stride && stride <= sbi->s_blocks_per_group)
 		ret = stride;
 	else
 		ret = 0;

2.10.2

_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

Thanks for the fixes! Your patches do fix my simple reproducer so that it gives me 2MiB aligned and size allocations for 64 MiB files, but when I run Xiong's xfstest I'm still getting misaligned allocations only for 64 MiB files. 32 MiB and 128 MiB sized files still work and give me a PMD.

I've pared down his xfstest to be a pretty minimal reproducer, and you can find it here:

https://git.kernel.org/cgit/linux/kernel/git/zwisler/xfstests-dev.git/log/?h=ext4_PMD_allocation

You can just revert the top patch in this tree (which basically just comments out 90% of the test and adds some debug) to get back to Xiong's full test.

The kernel tree I tested with today using your fixes can be found here:

https://git.kernel.org/cgit/linux/kernel/git/zwisler/linux.git/log/?h=ext4_PMD_allocation

I'm getting "No repositories found". And looking at the top level git.kernel.org, it doesn't look like there is an zwisler/xfstests-dev.git tree there at all. Was it uploaded correctly?

- Ted

The above link works for me, and I can see it in the top level view:

https://git.kernel.org/cgit/

I just created the repo today, though. Maybe it's just taking a mirror a second to catch up or something?

Please let me know if you continue to be unable to see it, and I'll figure it out.

- Ross

Yup, that must have been what it was. I must have caught a DNS mirror that was slow in updating.

- Ted