Skip to contentSkip to conversation

fscache review comments, part 1

DISCUSSION Christoph Hellwig 12 participants 23 messages Sep 28, 2006 – Oct 17, 2006 VFS

lore
Share & tools
lore
#0 lore raw

Just to be on the same boat, all these comments are against
http://people.redhat.com/~dhowells/nfs/nfs+fscache-16.tar.bz2

This mail is just the core-touching paches, the fscache and cachefiles code will get a review of it's own later on.

01-ino64.diff:
ACK

02-ino64-nfs.diff:
Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

03-fsmisc.diff:
ACK

05-release-page.diff:
ACK, though I'd wish someone could come up with nicer names for read_cache_pages_invalidate_page(s).

06-block-afs.diff:
ACK.

09-cachefiles-ia64.diff:
ACK. (Although I remember at some point I wanted to unify copy_page and copy_highpage)

11-autofs-dcache.diff:
ACK. Should probably go in ASAP.

12-reiserfs-dcache.diff:
ACK. Again should probably go in ASAP.

Note that reiserfs_kill_sb could be written a lot nicer by using a local variable.

static void reiserfs_kill_sb(struct super_block *sb)

{
	struct reiserfs_sb_info *si = REISERFS_SB(s);

	if (si) {
		if (si->xattr_root) {
			d_invalidate(si->xattr_root);
			dput(si->xattr_root);
		}
		if (si->priv_root) {
			d_invalidate(si->priv_root);
			dput(si->priv_root);
		}
	}

	kill_block_super(sb);
}

13-dcache-crunch.diff:
I know Jan has been rather unhappy with this, and I tend to agree to at least the code duplication part. I wish I had time to look into a better way to implement the rather nice idea behind shrink_dcache_for_umount.
Why is this patch required for fscache anyway?
(and why is it last in the series then?)

#1 lore raw
Quoted 7 lines · Christoph Hellwig

02-ino64-nfs.diff:
Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

A general question - is there a known list of common applications that have problems with 64-bit inodes? It's been mentioned several times, but I'm wondering if any effort is going toward fixing those apps so that 5 years from now (or whenever) we CAN have 64-bit inodes. If it relates to on-disk formats like ustar, we can always work with something like star to ensure that it is useful in the future.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

#2 lore raw
Quoted 9 lines · Andreas Dilger
Quoted 7 lines · Christoph Hellwig

02-ino64-nfs.diff:
Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

Quoted 6 lines

A general question - is there a known list of common applications that have problems with 64-bit inodes? It's been mentioned several times, but I'm wondering if any effort is going toward fixing those apps so that 5 years from now (or whenever) we CAN have 64-bit inodes. If it relates to on-disk formats like ustar, we can always work with something like star to ensure that it is useful in the future.

The most important ones we've seen at SGI were various version of tar and many commercial backup applications that silently (!) lost data on restore with 64bit inode numbers. I wasn't directly involved with those issue, but maybe some people that were at SGI at that time can fill in more details.

#3 lore raw
Quoted 16 lines · Christoph Hellwig
Quoted 8 lines · Andreas Dilger
Quoted 7 lines · Christoph Hellwig

02-ino64-nfs.diff:
Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

Quoted 6 lines

A general question - is there a known list of common applications that have problems with 64-bit inodes? It's been mentioned several times, but I'm wondering if any effort is going toward fixing those apps so that 5 years from now (or whenever) we CAN have 64-bit inodes. If it relates to on-disk formats like ustar, we can always work with something like star to ensure that it is useful in the future.

Quoted 5 lines

The most important ones we've seen at SGI were various version of tar and many commercial backup applications that silently (!) lost data on restore with 64bit inode numbers. I wasn't directly involved with those issue, but maybe some people that were at SGI at that time can fill in more details.

I wasn't involved with this much either, but I know people have cited Networker a number of times.

--Tim

#4 lore raw
In reply to Christoph Hellwig
Quoted 1 line · Christoph Hellwig

Why is this patch required for fscache anyway?

It's not, it's just there in my patchset because that's where I developed it. I should probably drop it from there as Andrew is now carrying it.

Quoted 1 line

(and why is it last in the series then?)

Because it is.

David

#5 lore raw
In reply to Christoph Hellwig
Quoted 6 lines · Christoph Hellwig

Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

This really needs fixing quite urgently. We've seen it break ld.so/libdl (and other things, but dynamic loading is one of the major pains as it's not entirely obvious).

David

#6 lore raw
Quoted 9 lines · On Mon, 02 Oct 2006 14:40:54 +0100 David Howells <[email protected]> wrote:
Quoted 6 lines · Christoph Hellwig

Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

Quoted 4 lines

This really needs fixing quite urgently. We've seen it break ld.so/libdl (and other things, but dynamic loading is one of the major pains as it's not entirely obvious).

Speaking of which.. I've been having struggling a bit with

vfs-make-filldir_t-and-struct-kstat-deal-in-64-bit-inode-numbers.patch nfs-represent-64-bit-fileids-as-64-bit-inode-numbers-on-32-bit-systems.patch

I don't have a good understanding of what the implications are for userspace. What are the risks and what are the advantages and what the level of urgency is behind those patches.

Help?

#7 lore raw
Quoted 17 lines · Andrew Morton
Quoted 9 lines · On Mon, 02 Oct 2006 14:40:54 +0100 David Howells <[email protected]> wrote:
Quoted 6 lines · Christoph Hellwig

Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

Quoted 4 lines

This really needs fixing quite urgently. We've seen it break ld.so/libdl (and other things, but dynamic loading is one of the major pains as it's not entirely obvious).

Quoted 4 lines

Speaking of which.. I've been having struggling a bit with

vfs-make-filldir_t-and-struct-kstat-deal-in-64-bit-inode-numbers.patch nfs-represent-64-bit-fileids-as-64-bit-inode-numbers-on-32-bit-systems.patch

Well, that's the two patches I've commented on above :)

Quoted 3 lines

I don't have a good understanding of what the implications are for userspace. What are the risks and what are the advantages and what the level of urgency is behind those patches.

vfs-make-filldir_t-and-struct-kstat-deal-in-64-bit-inode-numbers.patch is perfectly fine and harmless, seems fine for 2.6.19.

nfs-represent-64-bit-fileids-as-64-bit-inode-numbers-on-32-bit-systems.patch is quite risky as I mentioned so I'd drop it for the time beeing - we shouldn't put it in as long as we don't have a mount option for it.

#8 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig

02-ino64-nfs.diff:
Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

I don't think it belongs at VFS level. Sure, NFS might implement that, but... I'd still like to see real-life examples of such breakage before bothering with extra complexity.

#9 lore raw

64-bit inode number issues

In reply to Christoph Hellwig
Quoted 10 lines · Christoph Hellwig

01-ino64.diff:
ACK

02-ino64-nfs.diff:
Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

This problem doesn't just apply to NFS. It applies to any filesystem that can generate 64-bit inode numbers - which includes Ext3, I believe - so not applying the NFS patch doesn't really solve your problem (Al Viro asked me for examples of that, btw) and leaves my problem unsolved.

I think any solution has to apply at the VFS level or higher. I'm not sure Al agrees though. I think we have to do one of the following:

(1) Turn on inode number compression (which tries to hide the problem).

(2) Give an error to stat and getdents if the inode number can't be represented by 32-bits.

And I think we're going to do this, that we need to do this on a per-superblock basis or on a per-system basis, and I think it should be _off_ by default. It might also be possible to do it on a per-process level, but I suspect that'd be too much effort.

David

#10 lore raw

Re: 64-bit inode number issues

Quoted 2 lines · David Howells

This problem doesn't just apply to NFS. It applies to any filesystem that can generate 64-bit inode numbers - which includes Ext3, I believe

Actually, Ext3 can't do 64-bit inode numbers. I'd misremembered what Stephen told me.

David

#11 lore raw

Re: 64-bit inode number issues

Quoted 7 lines · David Howells
Quoted 2 lines · David Howells

This problem doesn't just apply to NFS. It applies to any filesystem that can generate 64-bit inode numbers - which includes Ext3, I believe

Actually, Ext3 can't do 64-bit inode numbers. I'd misremembered what Stephen told me.

Nope, we've thought about using 64-bit inode numbers for ext4, however --- it would solve a number of problems for us. Unfortunately, given issues with 64-bit inods at the VFS, glibc, and userspace layers, we decided it wasn't worth the pain and suffering that would be involved....

- Ted

#12 lore raw

Re: 64-bit inode number issues

Quoted 6 lines · Theodore Tso
Quoted 5 lines · David Howells
Quoted 2 lines · David Howells

This problem doesn't just apply to NFS. It applies to any filesystem that can generate 64-bit inode numbers - which includes Ext3, I believe

Quoted 8 lines
Quoted 2 lines

Actually, Ext3 can't do 64-bit inode numbers. I'd misremembered what Stephen told me.

Nope, we've thought about using 64-bit inode numbers for ext4, however --- it would solve a number of problems for us. Unfortunately, given issues with 64-bit inods at the VFS, glibc, and userspace layers, we decided it wasn't worth the pain and suffering that would be involved....

VFS is fine with those... glibc probably should be OK; userland might be nasty, though.

#13 lore raw

Re: 64-bit inode number issues

In reply to David Howells
Quoted 11 lines · David Howells
Quoted 6 lines

Unfortunately there's a lot of broken userspace that can't deal with 64bit inode numbers, so you need to make the lod behaviour a mount option at least, probably even the default. Given that we're going to run into problems like that it might make sense to make the option VFS-level instead of just in nfs. (Note:
XFS already has an option like that)

This problem doesn't just apply to NFS. It applies to any filesystem that can generate 64-bit inode numbers - which includes Ext3, I believe - so not applying the NFS patch doesn't really solve your problem (Al Viro asked me for examples of that, btw) and leaves my problem unsolved.

As you mention in your next mail ext3 is not affected. XFS for example is, that's why it has a 32bit inode numbers only mount option already.

Quoted 5 lines

I think any solution has to apply at the VFS level or higher. I'm not sure Al agrees though. I think we have to do one of the following:

(1) Turn on inode number compression (which tries to hide the problem).

Quoted 7 lines

(2) Give an error to stat and getdents if the inode number can't be represented by 32-bits.

And I think we're going to do this, that we need to do this on a per-superblock basis or on a per-system basis, and I think it should be _off_ by default. It might also be possible to do it on a per-process level, but I suspect that'd be too much effort.

This definitly should be on a per-superblock basis, aka the mount option
I suggested above. I don't think an actual scheme to deal with the
inode numbers at the VFS level makes sense, what I meant was only to do a vfs-level mount option (numerical) instead of a per-filesystem one.

Al commented he doesn't like that. He also think we should give the 64bit inode numbers a try, so I'd say:

o add an inode32 option for nfs that mirrors the XFS option
o turn it off by default in -mm and see what goes boom

#14 lore raw

Re: 64-bit inode number issues

Quoted 5 lines · Christoph Hellwig

Al commented he doesn't like that. He also think we should give the 64bit inode numbers a try, so I'd say:

o add an inode32 option for nfs that mirrors the XFS option
o turn it off by default in -mm and see what goes boom

I believe -mm is currently carrying the 64-bit inodes for NFS patch.

David

#15 lore raw

Re: 64-bit inode number issues

In reply to Christoph Hellwig
Quoted 3 lines · Christoph Hellwig

I don't think an actual scheme to deal with the
inode numbers at the VFS level makes sense, what I meant was only to do a vfs-level mount option (numerical) instead of a per-filesystem one.

Different fileystems might want different defaults - XFS (and every other filesystem) will still want 32 bit inodes by default right now, while NFS will probably want 64 bit inodes. That might complicate a vfs-level mount option, especially given that we don't want the XFS mount options to change.

Quoted 4 lines

Al commented he doesn't like that. He also think we should give the 64bit inode numbers a try, so I'd say:

o add an inode32 option for nfs that mirrors the XFS option

XFS uses an "inode64" option to turn on 64 bit inodes. If that is not specified it uses 32 bit inodes - there is no "inode32" option in XFS.

Quoted 1 line

o turn it off by default in -mm and see what goes boom

Will we even see the conditions for the NFS client to go boom in the environments -mm kernels are typically run? i.e. does someone have a test case that reliably triggers problems?

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#16 lore raw

Re: 64-bit inode number issues

Quoted 5 lines · David Chinner
Quoted 1 line

o turn it off by default in -mm and see what goes boom

Will we even see the conditions for the NFS client to go boom in the environments -mm kernels are typically run? i.e. does someone have a test case that reliably triggers problems?

I don't know about that, but we (Red Hat) have had customers logging the problem in our Bugzilla against NFS.

Coming up with a reliable test case is a little tricky, as it involves setting up a server to generate fileids > 0xffffffff, and manipulating inode numbers directly generally isn't allowed... Maybe it can be done with a ramfs hack.

David

#17 lore raw

Re: 64-bit inode number issues

Quoted 8 lines · David Howells
Quoted 5 lines · David Chinner
Quoted 1 line

o turn it off by default in -mm and see what goes boom

Will we even see the conditions for the NFS client to go boom in the environments -mm kernels are typically run? i.e. does someone have a test case that reliably triggers problems?

Quoted 6 lines

I don't know about that, but we (Red Hat) have had customers logging the problem in our Bugzilla against NFS.

Coming up with a reliable test case is a little tricky, as it involves setting up a server to generate fileids > 0xffffffff, and manipulating inode numbers directly generally isn't allowed... Maybe it can be done with a ramfs hack.

you can do a debug hack inside the kernel, where on 32 bit inode fs's you copy the lower 32 bit into the higher 32 bit (and maybe invert the bits or something); this can then be verified for all kinds of things all over the place ;)

Signature · 1 line
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
#18 lore raw

Re: 64-bit inode number issues

In reply to David Howells
Quoted 8 lines · David Howells
Quoted 5 lines · David Chinner
Quoted 1 line

o turn it off by default in -mm and see what goes boom

Will we even see the conditions for the NFS client to go boom in the environments -mm kernels are typically run? i.e. does someone have a test case that reliably triggers problems?

Quoted 6 lines

I don't know about that, but we (Red Hat) have had customers logging the problem in our Bugzilla against NFS.

Coming up with a reliable test case is a little tricky, as it involves setting up a server to generate fileids > 0xffffffff, and manipulating inode numbers directly generally isn't allowed... Maybe it can be done with a ramfs hack.

64 bit machine, >1TB sized XFS filesystem on the server, inode64 mount option, fill the filesystem to 99% full and then create lots of inodes. You should get inodes above 1TB then, and that will give you 64 bit inode numbers.

Typically the way we do this quickly is create an XFS filesystem then use the ag-wipe tool (*) we have in our QA suite that uses xfs_db to "fill" up all but the last AG in the filesystem by poking the right numbers into on disk structures. This is significantly faster than filling a terabyte of disk space.....

(*)http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-cmds/xfstests/tools/ag-wipe

FWIW, if you don't have more than a TB of disk space, then you might be able to create an XFS filesystem on loopback using a sparse file and then using the ag-wipe tool on the loopback filesystem.

HTH.

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#19 lore raw

Re: 64-bit inode number issues

In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

Al commented he doesn't like that. He also think we should give the 64bit inode numbers a try, so I'd say:

o add an inode32 option for nfs that mirrors the XFS option
o turn it off by default in -mm and see what goes boom

Unfortunately, that technique will not find any problems -- the people running systems with 64 bit inode numbers do not overlap with those running the -mm tree. The only real way to get any exercising of the relevant code paths would be to fake 64 bit inode numbers in a commonly used filesystem like ext3. The biggest difficultly with LFS has been that the API breakage is relatively silent; old applications work fine unless they encounter a large file, resulting in many lax applications.

-ben

Signature · 2 lines
"Time is of no importance, Mr. President, only life is important."
Don't Email: <[email protected]>.
#20 lore raw

Re: 64-bit inode number issues

Quoted 3 lines · Benjamin LaHaise

The only real way to get any exercising of the relevant code paths would be to fake 64 bit inode numbers in a commonly used filesystem like ext3.

Why not add a config option (or boot-time flash) or even mount option to have *existing* inode numbers shifted up 32-bits so that any userspace breakage will be detected really early on?

#21 lore raw
In reply to Christoph Hellwig
Quoted 3 lines · Christoph Hellwig

05-release-page.diff:
ACK, though I'd wish someone could come up with nicer names for read_cache_pages_invalidate_page(s).

They could always be nested:-). That way they don't pollute the namespace. This is one of the gcc extensions. If you do nm on the output, you see the nested function appear like this if not inlined:

	000000000000058f t invalidate_page.17245

That above derives from this:

int read_cache_pages(struct address_space *mapping, struct list_head *pages,
int (*filler)(void *, struct page *), void *data)

{
	struct page *page;
	struct pagevec lru_pvec;
	int ret = 0;

	/* see if a page needs releasing upon read_cache_pages() failure */
	void invalidate_page(struct address_space *mapping, struct page *page)
	{
		if (PagePrivate(page)) {
			if (TestSetPageLocked(page))
				BUG();
			page->mapping = mapping;
			do_invalidatepage(page, 0);
			page->mapping = NULL;
			unlock_page(page);
		}
		page_cache_release(page);
	}

	/* release a list of pages, invalidating them first if need be */
	void invalidate_pages(struct address_space *mapping,
			      struct list_head *pages)
	{
		struct page *victim;

		while (!list_empty(pages)) {
			victim = list_to_page(pages);
			list_del(&victim->lru);
			invalidate_page(mapping, victim);
		}
	}

	pagevec_init(&lru_pvec, 0);

	while (!list_empty(pages)) {
		page = list_to_page(pages);
		list_del(&page->lru);
		if (add_to_page_cache(page, mapping, page->index, GFP_KERNEL)) {
			invalidate_page(mapping, page);
			continue;
		}
		ret = filler(data, page);
		if (!pagevec_add(&lru_pvec, page))
			__pagevec_lru_add(&lru_pvec);
		if (ret) {
			invalidate_pages(mapping, pages);
			break;
		}
	}
	pagevec_lru_add(&lru_pvec);
	return ret;
}

David

#22 lore raw
Quoted 6 lines · David Howells
Quoted 3 lines · Christoph Hellwig

05-release-page.diff:
ACK, though I'd wish someone could come up with nicer names for read_cache_pages_invalidate_page(s).

Quoted 7 lines

They could always be nested:-). That way they don't pollute the namespace. This is one of the gcc extensions. If you do nm on the output, you see the nested function appear like this if not inlined:

	000000000000058f t invalidate_page.17245

That above derives from this:

This is C, not pascal - please don't do such fugly things..