Process hangs copying large file to cifs filesystem
This person's CIFS hang is happening because the hlist_unhashed() test in __mark_inode_dirty() is returning true, so the inode never gets itself onto sb->s_dirty. The writeback paths have no way to write out the pagecache.
a) Why are CIFS inodes not hashed?
b) Why does __mark_inode_dirty() skip unhashed inodes anyway?
Taking the hlist_unhashed() test out fixed it up, although CIFS only wants to write stuff out at 2 Mbytes/sec on 100bT, whereas NFS does wirespeed...
Begin forwarded message:
Date: Wed, 23 Jun 2004 12:44:54 +0100
From: Nuno Ferreira <[email protected]>
To: Jens Axboe <[email protected]>
Cc: Linux Kernel <[email protected]>
Subject: Re: Process hangs copying large file to cifs filesystem
Quoted 11 lines · Nuno Ferreira
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Nuno Ferreira
Quoted 5 lines · Nuno Ferreira
Hi,
I'm trying to copy a large file (200Mb or bigger) from an ext3 filesystem to a windows share mounted using CIFS and the cp process hangs, sometimes for a long time (several minutes). Calling ps, I can see that it's blocking onblk_congestion_wait.[...]
Quoted 2 lines
A sysrq-t back trace of that process would be interesting to see.
Quoted 6 lines · Andrew Morton
This person's CIFS hang is happening because the
hlist_unhashed()test in__mark_inode_dirty()is returning true, so the inode never gets itself ontosb->s_dirty. The writeback paths have no way to write out the pagecache.a) Why are CIFS inodes not hashed?
Presumably because it never does hash lookups. I wonder where it gets inumbers from, though - keeping them unique would at least require being able to see what's already in-core.
Quoted 1 line
b) Why does
__mark_inode_dirty()skip unhashed inodes anyway?
Ask Linus. I've always treated that as a behaviour filesystems might be using, and IIRC in some cases that was deliberately used. I have my suspicions about the origin of that animal, but that was before my time (during 2.1.50ish rewrite of inode.c) and inode.c is one place I try to touch as rarely as possible...
Quoted 7 lines · [email protected] wrote:
Quoted 5 lines · Andrew Morton
This person's CIFS hang is happening because the
hlist_unhashed()test in__mark_inode_dirty()is returning true, so the inode never gets itself ontosb->s_dirty. The writeback paths have no way to write out the pagecache.
Quoted 5 lines
Quoted 1 line
a) Why are CIFS inodes not hashed?
Presumably because it never does hash lookups. I wonder where it gets inumbers from, though - keeping them unique would at least require being able to see what's already in-core.
hm. Well cifs inodes seem to have unique i_ino's somehow.
Quoted 7 lines
Quoted 1 line
b) Why does
__mark_inode_dirty()skip unhashed inodes anyway?Ask Linus. I've always treated that as a behaviour filesystems might be using, and IIRC in some cases that was deliberately used. I have my suspicions about the origin of that animal, but that was before my time (during 2.1.50ish rewrite of
inode.c) andinode.cis one place I try to touch as rarely as possible...
Linus, why does __mark_inode_dirty() skip unhashed inodes?
Quoted 2 lines · Andrew Morton
Linus, why does
__mark_inode_dirty()skip unhashed inodes?
Hah. I wouldn't remember for the life of me. There could be multiple reasons, all of which I've forgotten:
- ramfs deadlocks on "sync()", since it can't write out inodes
- stale and bad inodes get unhashed, and end up being unwritable.
- totally stale reason that doesn't exist any more.
I suspect #3 migth well be true, but it's a scary change to try.
Anybody else remember better than me?
Linus
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Andrew Morton
Linus, why does
__mark_inode_dirty()skip unhashed inodes?Hah. I wouldn't remember for the life of me. There could be multiple reasons, all of which I've forgotten:
- ramfs deadlocks on "
sync()", since it can't write out inodes
hm. ramfs oopses on null ->writepage without the check. I'll fix that up.
Quoted 1 line
- stale and bad inodes get unhashed, and end up being unwritable.
Maybe we should check inode sanity in the writing path, rather than the dirtying path. We already check is_bad_inode() in write_inode().
Quoted 3 lines
- totally stale reason that doesn't exist any more.
I suspect #3 migth well be true, but it's a scary change to try.
It is indeed scary, yes. I think the path of least resistance here is for cifs to start hashing dem inodes.
A message that is not in the archive (sent off-list or not yet received): OF980F9E62.86CA6214-ON87256EBE.007E2269-86256EBE.007E1390@us.ibm.com
Quoted 8 lines · Steven French
Quoted 2 lines
It is indeed scary, yes. I think the path of least resistance here is
for
Quoted 1 line
cifs to start hashing dem inodes.
Yes, agreed. I will test that minor change, to call
insert_inode_hashto new cifs inodes, this weekend.
Great, thanks. Other things:
- cifs_commit_write() should be running set_page_dirty() in the !PageUptodate() case as well, yes?
- Currently, a cifs set_page_dirty() falls into __set_page_dirty_buffers(). That's OK, but incurs more locking than is needed.
Suggest you make cifs_addr_ops.set_page_dirty point at __set_page_dirty_nobuffers().
Quoted 6 lines · Linus Torvalds
- ramfs deadlocks on "
sync()", since it can't write out inodes- stale and bad inodes get unhashed, and end up being unwritable.
- totally stale reason that doesn't exist any more.
There might also be a race on "get rid of inode" from the filesystem standpoint, and unhashing the inode would be the atomic "now nobody will call us any more" point. NFS exports etc in particular tend to have all these interesting things where kNFSd ends up touching inodes even after they have been unliked, because it uses the magic "look up by inode" interace.
Although I think the I_CLEARING bits etc should also act as atomic points of "nobody will touch this any more", but what do I know?
Anyway, I do believe that if we remove the check for hashedness, we'd need to really stress it over a long period in 2.7-like circumstances before we do it in a "real" kernel.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds
- ramfs deadlocks on "
sync()", since it can't write out inodes- stale and bad inodes get unhashed, and end up being unwritable.
Quoted 8 lines
Quoted 1 line
- totally stale reason that doesn't exist any more.
There might also be a race on "get rid of inode" from the filesystem standpoint, and unhashing the inode would be the atomic "now nobody will call us any more" point. NFS exports etc in particular tend to have all these interesting things where kNFSd ends up touching inodes even after they have been unliked, because it uses the magic "look up by inode" interace.
hm, but we've called ->dirty_inode() on it prior to testing the unhashedness.
Quoted 2 lines
Although I think the
I_CLEARINGbits etc should also act as atomic points of "nobody will touch this any more", but what do I know?
That should be so.
Quoted 4 lines
Anyway, I do believe that if we remove the check for hashedness, we'd need to really stress it over a long period in 2.7-like circumstances before we do it in a "real" kernel.
Yes, this will explode in our faces for sure. I was going to stick a printk in there as a starting point, but ramfs might be a problem. It would, I think, be cleaner to permit ramfs inodes to be dirtied, but refuse to write them out due to memory-backedness. I'll play with it a bit.