fiemap is broken for sparse file in ext4?

DISCUSSION Tao Ma 3 participants 9 messages Jun 11, 2010

Share & tools

Tao Ma

author

2010-06-11 02:16 UTC #0 lore raw

Hi Ted and other ext4 gurus,
I found fiemap may be broken for sparse files in ext4. Here is a simple
example.

dd if=/dev/zero of=testfile1 bs=1M count=1
using fiemap shows that it has a delalloc extent.
Logical: 0 Ext length: 1048576 Physical: 0 flags: 7

flags 7 means FIEMAP_EXTENT_LAST, FIEMAP_EXTENT_UNKNOWN and FIEMAP_EXTENT_DELALLOC,

while if we create a sparse file, fiemap will not show the delalloc extent.
dd if=/dev/zero of=testfile1 bs=1M count=1 seek=1
using fiemap shows that it has no extent for the file. while we should have some output like:
Logical: 1048576 Ext length: 1048576 Physical: 0 flags: 7

So we have different output with sparse and non-sparse file. Is it a bug for ext4?

Regards,
Tao

Eric Sandeen

2010-06-11 04:08 UTC #1 lore raw

Quoted 11 lines · Tao Ma

Hi Ted and other ext4 gurus,
I found fiemap may be broken for sparse files in ext4. Here is a
simple example.
dd if=/dev/zero of=testfile1 bs=1M count=1
using fiemap shows that it has a delalloc extent.
Logical: 0 Ext length: 1048576 Physical: 0 flags: 7
flags 7 means FIEMAP_EXTENT_LAST, FIEMAP_EXTENT_UNKNOWN and FIEMAP_EXTENT_DELALLOC,

Quoted 8 lines

while if we create a sparse file, fiemap will not show the delalloc extent. dd if=/dev/zero of=testfile1 bs=1M count=1 seek=1
using fiemap shows that it has no extent for the file. while we should have some output like:
Logical: 1048576 Ext length: 1048576 Physical: 0 flags: 7
So we have different output with sparse and non-sparse file. Is it a bug for ext4?

What are you using to call fiemap? Here it seems to be working:

# dd if=/dev/zero of=testfile1 bs=1M count=1 seek=1;

# filefrag -v testfile1
Filesystem type is: ef53
Filesystem cylinder groups is approximately 119
File size of testfile1 is 2097152 (512 blocks, blocksize 4096)
 ext logical physical expected length flags
   0     256   151946               1 merged
   1     257   151951   151946      2 merged
   2     259   152434   151952    253 merged,eof
testfile1: 4 extents found

And with my homegrown fiemap tester:

# ./fiemap-test testfile1
ext:   0 logical: [     256..     256] phys:   151946..  151946 flags: 0x1000 tot: 1
ext:   1 logical: [     257..     258] phys:   151951..  151952 flags: 0x1000 tot: 2
ext:   2 logical: [     259..     511] phys:   152434..  152686 flags: 0x1001 tot: 253

... seems ok here, aside from the fragmentation :) ...

-Eric

Tao Ma

author

2010-06-11 05:06 UTC #2 lore raw

Hi Eric,
Thanks for the quick response.

Quoted 18 lines · Eric Sandeen

Quoted 17 lines · Tao Ma
Hi Ted and other ext4 gurus,
I found fiemap may be broken for sparse files in ext4. Here is a
simple example.
dd if=/dev/zero of=testfile1 bs=1M count=1
using fiemap shows that it has a delalloc extent.
Logical: 0 Ext length: 1048576 Physical: 0 flags: 7
flags 7 means FIEMAP_EXTENT_LAST, FIEMAP_EXTENT_UNKNOWN and FIEMAP_EXTENT_DELALLOC,
while if we create a sparse file, fiemap will not show the delalloc extent. dd if=/dev/zero of=testfile1 bs=1M count=1 seek=1
using fiemap shows that it has no extent for the file. while we should have some output like:
Logical: 1048576 Ext length: 1048576 Physical: 0 flags: 7

Quoted 4 lines

Quoted 2 lines
So we have different output with sparse and non-sparse file. Is it a bug for ext4?
What are you using to call fiemap? Here it seems to be working:

I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)

Quoted 12 lines

# dd if=/dev/zero of=testfile1 bs=1M count=1 seek=1;

# filefrag -v testfile1
Filesystem type is: ef53
Filesystem cylinder groups is approximately 119
File size of testfile1 is 2097152 (512 blocks, blocksize 4096)
  ext logical physical expected length flags
    0     256   151946               1 merged
    1     257   151951   151946      2 merged
    2     259   152434   151952    253 merged,eof
testfile1: 4 extents found

I guess maybe filefrag use the diffrent ioctl flag, maybe FIEMAP_FLAG_SYNC to let ext4 sync first.

Regards,
Tao

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <stdint.h>
#include <inttypes.h>
#include <string.h>

#include "fiemap.h"

#define FS_IOC_FIEMAP _IOWR('f', 11, struct fiemap)

static unsigned int num_extents = 1024;
static unsigned int estimate_extents = 1;

static unsigned long long map_start = 0ULL; static unsigned long long map_len = FIEMAP_MAX_OFFSET; static char *fname;
static unsigned int blocksize = 4096;

static void usage(void)

{
	printf("Usage: fiemap filename\n");
	exit(1);
}

static void print_extent(struct fiemap_extent *extent)

{
	__u64 val;

	printf("Logical: ###[%8"PRIu64"]\t", extent->fe_logical);
	printf("Ext length: ###[%8"PRIu64"]\t", extent->fe_length);
	printf("Physical: ###[%8"PRIu64"]\t", extent->fe_physical);
	printf("flags: %u\t", extent->fe_flags);

	if (extent->fe_flags & FIEMAP_EXTENT_SHARED)
		printf("Yeah, found an shared extent\n");
	if (extent->fe_flags & FIEMAP_EXTENT_UNWRITTEN)
		printf("Yeah, you are hole?\n");

	printf("\n");
}

static void show_results(struct fiemap *fiemap)

{
	int i;

	printf("Extents returned: %u\n", fiemap->fm_mapped_extents);
	if (fiemap->fm_extent_count == 0 || fiemap->fm_mapped_extents == 0)
		return;

	for (i = 0; i < fiemap->fm_mapped_extents; i++)
		print_extent(&fiemap->fm_extents[i]);

}

static int
figure_extents(int fd, unsigned int *num)

{
	int ret;
	struct fiemap fiemap = { 0, };

	fiemap.fm_start = map_start;
	fiemap.fm_length = map_len;
	fiemap.fm_flags = 0;

	ret = ioctl(fd, FS_IOC_FIEMAP, &fiemap);
	if (ret == -1) {
		fprintf(stderr, "fiemap get count error %d: \"%s\"\n", errno,
			strerror(errno));
		return -1;
	}

	*num = fiemap.fm_mapped_extents;

	return 0;
}

int
main(int argc, char **argv)

{
	int ret, fd;
	struct fiemap *fiemap;

	fd = open(argv[1], O_RDONLY);
	if (fd == -1) {
		fprintf(stderr, "open error %d: \"%s\"\n", errno,
			strerror(errno));
		return -1;
	}

	if (figure_extents(fd, &num_extents))
		return -1;

	printf("Extents in file \"%s\":    %u\n", argv[1], num_extents);

	fiemap = malloc(sizeof(fiemap) +
			num_extents * sizeof(struct fiemap_extent));
	if (fiemap == NULL) {
		fprintf(stderr, "malloc error %d: \"%s\"\n", errno,
			strerror(errno));
		return -1;
	}

	fiemap->fm_start = map_start;
	fiemap->fm_length = map_len;
	fiemap->fm_extent_count = num_extents;

	ret = ioctl(fd, FS_IOC_FIEMAP, fiemap);
	if (ret == -1) {
		if (errno == EBADR) {
			fprintf(stderr, "Kernel does not support the flags: 0x%x\n",
				fiemap->fm_flags);
			return -1;
		}
		fprintf(stderr, "fiemap error %d: \"%s\"\n", errno,
			strerror(errno));
		return -1;
	}

	show_results(fiemap);

	close(fd);

	return 0;
}

Eric Sandeen

2010-06-11 13:43 UTC #3 lore raw

Quoted 4 lines · Tao Ma

Hi Eric,
Thanks for the quick response.
On 06/11/2010 12:08 PM, Eric Sandeen wrote:

...

Quoted 16 lines

Quoted 1 line
What are you using to call fiemap? Here it seems to be working:
I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)
Quoted 12 lines
# dd if=/dev/zero of=testfile1 bs=1M count=1 seek=1;
# filefrag -v testfile1
Filesystem type is: ef53
Filesystem cylinder groups is approximately 119
File size of testfile1 is 2097152 (512 blocks, blocksize 4096)
  ext logical physical expected length flags
    0     256   151946               1 merged
    1     257   151951   151946      2 merged
    2     259   152434   151952    253 merged,eof
testfile1: 4 extents found

Quoted 2 lines

I guess maybe filefrag use the diffrent ioctl flag, maybe FIEMAP_FLAG_SYNC to let ext4 sync first.

My tester isn't calling sync or using the sync flag, AFAIK. I'll take a look at yours.

Thanks,
-Eric

Quoted 3 lines

Regards,
Tao

Eric Sandeen

2010-06-11 15:46 UTC #4 lore raw

In reply to Tao Ma

Quoted 4 lines · Tao Ma

Quoted 1 line
What are you using to call fiemap? Here it seems to be working:
I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)

OK I take it back, I do see it. *cough* too many different filesystems on this box ... ;)

Yes, it does look like a bug.

-Eric

Greg Freemyer

2010-06-11 16:46 UTC #5 lore raw

Quoted 7 lines · Eric Sandeen

Quoted 4 lines · Tao Ma
Quoted 1 line
What are you using to call fiemap? Here it seems to be working:
I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)

Quoted 6 lines

OK I take it back, I do see it. *cough* too many different filesystems on this box ... ;)
Yes, it does look like a bug.
-Eric

I'm not clear how this is supposed to work.

Is this just a userspace bug in that FIEMAP_FLAG_SYNC is needed in the fiemap call, or is there a kernel bug as well?

If fiemap is called with the FIEMAP_FLAG_SYNC flag is there currently a problem?

Thanks
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Eric Sandeen

2010-06-11 16:56 UTC #6 lore raw

Quoted 10 lines · Greg Freemyer

Quoted 9 lines · Eric Sandeen
Quoted 4 lines · Tao Ma
Quoted 1 line
What are you using to call fiemap? Here it seems to be working:
I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)
OK I take it back, I do see it. *cough* too many different filesystems on this box ... ;)

Quoted 8 lines

Quoted 3 lines
Yes, it does look like a bug.
-Eric
I'm not clear how this is supposed to work.
Is this just a userspace bug in that FIEMAP_FLAG_SYNC is needed in the fiemap call, or is there a kernel bug as well?

The flag is optional, though maybe filefrag should use it.

Without it, we should get the proper logical offset and a delalloc-flagged extent returned

Quoted 1 line

If fiemap is called with the FIEMAP_FLAG_SYNC flag is there currently a problem?

yep, we should get one delalloc extent in the results and we don't.

-Eric

Quoted 2 lines

Thanks
Greg

Tao Ma

author

2010-06-11 23:47 UTC #7 lore raw

Hi Eric,

Quoted 28 lines · Eric Sandeen

Quoted 14 lines · Greg Freemyer
Quoted 12 lines · Eric Sandeen
Quoted 6 lines · Tao Ma
Quoted 2 lines
What are you using to call fiemap? Here it seems to be working:
I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)
OK I take it back, I do see it. *cough* too many different filesystems on this box ... ;)
Quoted 9 lines
Quoted 4 lines
Yes, it does look like a bug.
-Eric
I'm not clear how this is supposed to work.
Is this just a userspace bug in that FIEMAP_FLAG_SYNC is needed in the fiemap call, or is there a kernel bug as well?
The flag is optional, though maybe filefrag should use it.

Quoted 9 lines

Without it, we should get the proper logical offset and a delalloc-flagged extent returned
Quoted 2 lines
If fiemap is called with the FIEMAP_FLAG_SYNC flag is there currently a problem?
yep, we should get one delalloc extent in the results and we don't.

With FIEMAP_FLAG_SYNC set, my test shows that we get one extent without dealloc flags. So no problem with it. We have another different test result? ;)

Regards,
Tao

Eric Sandeen

2010-06-11 23:55 UTC #8 lore raw

Quoted 40 lines · Tao Ma

Hi Eric,
Quoted 38 lines · Eric Sandeen
Quoted 15 lines · Greg Freemyer
Quoted 12 lines · Eric Sandeen
Quoted 6 lines · Tao Ma
Quoted 2 lines
What are you using to call fiemap? Here it seems to be working:
I just wrote a simple test program by calling ioctl. It is attached. btw, you need to call it immediately after dd so that we have a chance that ext4 don't have time to allocate extents. ;)
OK I take it back, I do see it. *cough* too many different filesystems on this box ... ;)
Quoted 9 lines
Quoted 4 lines
Yes, it does look like a bug.
-Eric
I'm not clear how this is supposed to work.
Is this just a userspace bug in that FIEMAP_FLAG_SYNC is needed in the fiemap call, or is there a kernel bug as well?
The flag is optional, though maybe filefrag should use it.
Without it, we should get the proper logical offset and a delalloc-flagged
extent returned
Quoted 3 lines
If fiemap is called with the FIEMAP_FLAG_SYNC flag is there currently a problem?

Quoted 5 lines

Quoted 2 lines
yep, we should get one delalloc extent in the results and we don't.
With FIEMAP_FLAG_SYNC set, my test shows that we get one extent without dealloc flags. So no problem with it.
We have another different test result? ;)

No, I misread. :( Calling with FIEMAP_FLAG_SYNC works as expected.

-Eric

Quoted 3 lines

Regards,
Tao