Skip to contentSkip to conversation

How does ext2 implement sparse files?

DISCUSSION Lars Noschinski 7 participants 7 messages Jan 31, 2008 – Feb 1, 2008

lore
Share & tools
lore
#0 lore raw

Hello!

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

Any hints would be greatly appreciated,
Lars.

[Please CC me on answers, I'm not subscribed]

#1 lore raw
Quoted 10 lines · Lars Noschinski

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

Any hints would be greatly appreciated,

While I don't actually know, I always thought it was up to the application to create files with holes by seeking over the holes before writing data. That seems like the simplest way to me.

--
Len Sorensen

#2 lore raw
In reply to Lars Noschinski
Quoted 11 lines · Lars Noschinski

Hello!

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

In ext2 (and most other block filesystems) all files are sparse files. If you write to an address in the file for which no block is allocated, the filesystem allocates a block and writes the contents to disk, regardless of whether that block is at the end of the file (the usual case of lengthening a non-sparse file), in the middle of the file (filling in holes in a sparse file), or past the the end of the file (making a file sparse).

-- Chris

#3 lore raw
In reply to Lars Noschinski
Quoted 8 lines · Lars Noschinski

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

It does not explicitely detect holes; holey data is just never written so no space for it is allocated.

-Andi

#4 lore raw
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Lars Noschinski

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

Quoted 7 lines
Quoted 3 lines

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

It does not explicitely detect holes; holey data is just never written so no space for it is allocated.

does anybody know how to make a hole in a large file which already has real content from user space application?
In my project I need this function to delete a piece of content from an exist large effectively.
thanks.

Quoted 8 lines

-Andi

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
Quoted 19 lines · Shuduo Sang
Quoted 8 lines · Andi Kleen
Quoted 5 lines · Lars Noschinski

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

Quoted 8 lines
Quoted 4 lines

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

It does not explicitely detect holes; holey data is just never written so no space for it is allocated.

Quoted 7 lines

does anybody know how to make a hole in a large file which already has real content from user space application?
In my project I need this function to delete a piece of content from an exist large effectively.
thanks.

Some OSes use fcntl() F_FREESP/F_FREESP64 to be able to free allocated space in files (ie make holes if supported by underlying fs)

AFAIK, linux can generically do this only at the end of a file (ftruncate()) , not at random points.

XFS has special support for FREESP (it comes from IRIX), implemented as an ioctl()

Check for XFS_IOC_FREESP and XFS_IOC_UNRESVSP in fs/xfs/xfs_fs.h

#6 lore raw
In reply to Andi Kleen

[Sorry for using an invalid mail address in my previous post]

Quoted 7 lines · Andi Kleen
Quoted 5 lines · Lars Noschinski

For an university project, we had to write a toy filesystem (ext2-like), for which I would like to implement sparse file support. For this, I digged through the ext2 source code; but I could not find the point, where ext2 detects holes.

Quoted 6 lines
Quoted 3 lines

As far as I can see from fs/buffer.c, an hole is a buffer_head which is not mapped, but uptodate. But I cannot find a relevant source line, where ext2 makes usage of this information.

It does not explicitely detect holes; holey data is just never written so no space for it is allocated.

Thanks for the answers. No that I know that, it is kind of obvious. I was irritated yesterday by a bug a bug in our implementation (number of used blocks is reported as filesize/blocksize); so I was expecting the VFS to issue write requests for empty pages (which would be sillly, I know).

Thanks,
Lars.