fs: cachefiles: Add support for large files in filesystem caching

The below patch which was first posted here:

   https://lkml.org/lkml/2012/4/3/193

and tested by me and picked up by the maintainer here:

   https://lkml.org/lkml/2012/5/1/57

But it seems to not to have made to the linux-next or Linus tree. I'm resending what it seems like a lost patch.

From: Justin Lecher <[email protected]>

Support the caching of large files.

https://bugzilla.kernel.org/show_bug.cgi?id=31182

Signed-off-by Justin Lecher <[email protected]>

Tested-by Suresh Jayaraman <[email protected]>

---
 fs/cachefiles/rdwr.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

fs/cachefiles/rdwr.c +0 -0

index 0e3c092..d136686 100644

Quoted 12 lines

From: Justin Lecher <[email protected]>

Support the caching of large files.

https://bugzilla.kernel.org/show_bug.cgi?id=31182

Signed-off-by Justin Lecher <[email protected]>

Tested-by Suresh Jayaraman <[email protected]>

---
 fs/cachefiles/rdwr.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

fs/cachefiles/rdwr.c int cachefiles_write_page(struct fscache_storage *op, struct page *page) · line 922

Quoted 10 lines

fs/cachefiles/rdwr.c +1 -1

index 0e3c092..d136686 100644
@@ -919,7 +919,7 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page)
 	dget(object->backer);
 	mntget(cache->mnt);
 	file = dentry_open(object->backer, cache->mnt, O_RDWR,
-			   cache->cache_cred);
+			   (O_RDWR | O_LARGEFILE), cache->cache_cred);

Doesn't compile.

How extensively was this change tested? Please describe the testing which was performed?

Quoted 3 lines

How extensively was this change tested? Please describe the testing which was performed?

The original patch was tested by mounting a NFS share with fscache option enabled, do a md5sum on a nfs file larger than 2GB and ensure that the file is getting cached by fscache (watch cache size growing) on x86_64.

Quoted 4 lines · On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:

Quoted 3 lines

How extensively was this change tested? Please describe the testing which was performed?

Quoted 4 lines

The original patch was tested by mounting a NFS share with fscache option enabled, do a md5sum on a nfs file larger than 2GB and ensure that the file is getting cached by fscache (watch cache size growing) on x86_64.

Well it will need a lot more coverage testing than that. truncate? expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), behavior at the new max file size (what is that?), etc.

4 Gb on a daily bases. This isn't a testsuite, but it worked fine for

Quoted 4 lines · On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:

Quoted 3 lines

How extensively was this change tested? Please describe the testing which was performed?

Quoted 4 lines

The original patch was tested by mounting a NFS share with fscache option enabled, do a md5sum on a nfs file larger than 2GB and ensure that the file is getting cached by fscache (watch cache size growing) on x86_64.

Well it will need a lot more coverage testing than that. truncate? expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), behavior at the new max file size (what is that?), etc.

Quoted 6 lines · Andrew Morton

Quoted 4 lines · On Tue, 19 Jun 2012 10:57:27 +0530 Suresh Jayaraman <[email protected]> wrote:

Quoted 3 lines

How extensively was this change tested? Please describe the testing which was performed?

Quoted 9 lines

Quoted 4 lines

The original patch was tested by mounting a NFS share with fscache option enabled, do a md5sum on a nfs file larger than 2GB and ensure that the file is getting cached by fscache (watch cache size growing) on x86_64.

Well it will need a lot more coverage testing than that. truncate? expanding truncate? write, lseek, pwrite, pread(), mmap(MAP_SHARED), behavior at the new max file size (what is that?), etc.

The changelog could be slightly misleading as the patch doesn't really add LFS support. It just passes O_LARGEFILE flag to dentry_open() to make VFS allow cachefiles to open larger files (instead of failing it with -EOVERFLOW). Cachefiles is just a cache that uses the VFS/VM interfaces to get another disk filesystem to do the requisite I/O on its behalf. So, I doubt whether more coverage testing is required for this change.

Anyway, I'm attaching a quick and dirty test program that might help (not well tested though). I'm unable to to test it as I have modified my setup and have to redo it.

Justin, do you have the setup and can you get this tested?

/*
* Few tests to ensure LFS (Large File Support) is working correctly. *
* The requests (seek, truncate, write, mmap etc) that extend beyond the offset
* maximum (off_t) should succeed.
*
* Compile it with -D_FILE_OFFSET_BITS=64
*
* Run it like:
* ./lfs-tests large_file
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>

#define BUF_SZ 32

#define err(msg) \
do { perror(msg); exit(EXIT_FAILURE); } while (0)

void report_size(int fd, struct stat *st)

{
	fstat(fd, st);
	printf("size of file: %ld\n", st->st_size);

	return;
}

int main(int argc, char *argv[])

{
	int fd;
	int rc = 0;
	unsigned long offset = 0;
	struct stat stbuf;
	char buf[BUF_SZ];
	void *file_mem;

	if (argc != 2) {
		fprintf(stderr, "Usage: %s test_file\n", argv[0]);
		exit(EXIT_FAILURE);
	}

	fd = open(argv[1], O_RDWR | O_APPEND);
	if (fd == -1)
		err("open");

	memset(buf, 0, sizeof(buf));

	/* seek to the end of file */
	offset = lseek(fd, 0, SEEK_END);
	if (offset < 0)
		err("lseek");

	/* seek past the end of file */
	offset = lseek(fd, BUF_SZ, SEEK_END);
	printf("%ld\n", offset);
	if (offset < 0)
		err("lseek");

	/* truncate it to < 2GB and report size */
	rc = ftruncate(fd, 2147483648 - BUF_SZ);
	if (rc != 0)
		err("ftruncate");

	report_size(fd, &stbuf);

	/* truncate it to a size larger than 2GB */
	rc = ftruncate(fd, 4294967360);
	if (rc != 0)
		err("ftruncate");

	report_size(fd, &stbuf);

	/* seek to the end of file */
	offset = lseek(fd, 0, SEEK_END);
	if (offset < 0)
		err("lseek");

	memset(buf, 'a', sizeof(buf));

	/* write past 4GB */
	rc = write(fd, buf, sizeof(buf));
	if (rc == -1)
		err("write");

	report_size(fd, &stbuf);

	offset = lseek(fd, 0, SEEK_END);
	if (offset < 0)
		err("lseek");

	/* pwrite and pread */
	rc = pread(fd, buf, BUF_SZ, offset);
	if (rc == -1)
		err("pread");

	rc = pwrite(fd, buf, sizeof(buf), offset);
	if (rc == -1) 
		err("pwrite");

	report_size(fd, &stbuf);

	/* page align offset */
	offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1);

	file_mem = mmap(0, BUF_SZ, PROT_WRITE, MAP_SHARED, fd, offset);
	if (file_mem == MAP_FAILED)

		err("mmap");

	munmap(file_mem, 64);

	close(fd);

	return EXIT_SUCCESS;
}