hfsplus: fixes worst-case unicode to char conversion of file names and attributes
------------------------------
Quoted 14 lines · BST Vyacheslav Dubeyko
On Apr 8, 2014, at 9:09 PM, Anton Altaparmakov wrote:
[snip]
Quoted 8 lines
Unfortunately, it doesn't make sense to have fix only in
hfsplus_listxattr(). It fixes nothing for xattr names. Of course, you can mention name of the patch. :) But such partial fix doesn't make sense for me. It will be more clear way to have one patch for file names and another patch for xattr names.You are entitled to your opinion but I agree with Hin-Tak. This makes perfect sense as it fixes the use of a function which can otherwise cause an erroneous EIO error to be returned from listxattr(). This in itself is a worthwhile fix. Your idea of having to fix everything or nothing at all is frankly quite silly and would mean that hardly anything ever gets fixed... Fixing small problems individually is just fine, especially when they are such obvious ones.
Quoted 6 lines
I have another opinion. :) But, of course, it is possible to see on things in different ways. I don't think that topic of splitting fixes between patches is really important. But my suggestion has more logical basis.
But, anyway, I suggest to use kmem cache for the case of xattr names instead of kmalloc.
This patch fix usages of hfsplus_uni2asc(). There are only two.
I think we all agree that there is a lot more work to be done in the attribute side to make sure that getting and setting long non-english attributes to work. And there will be a series of patches to it. The discussion is really on how this is split.
There are two uses of HFSPLUS_MAX_STRLEN concern asc2uni(), and are correct where they are. There is one other use in
super.c:319: buf->f_namelen = HFSPLUS_MAX_STRLEN;
That may need some thinking. It is probably correct.
For HFSPLUS_ATTR_MAX_STRLEN, besides the one I changed, there are 9 other uses as size of buffers (and a similar number in checks for length over-run, etc).
As I wrote earlier, some of these are setters, rather than getters. Setters and getters are different. The setters probably should use a large enough buffer according to userland, (ie. not a file-system specific one), and let the lower-layer of the file system returns an error when one is trying to set a too-long attribute. The getters needs to cope with what listxttr() returns.
So I see there should be at least two forth-coming patches related to attributes. One can bundle the attribute part of the current patch to the latter; but then that's assuming that the getter part of these further patches is logically one. It may be complex enough to require splitting up. We have 9 buffers, divisible at least now into two groups, getters and setters - it might need to be divided further. There are OS-prefixes complications along the way, so we may end up having to do 3-4 additional patches.
At this point it is not obvious that half of the current patch should be deferred, and where/what it should defer to. So I am thinking just "commit early, commit often".
Retrospectively - when we look back on it afterwards, in a few month's time -, the whole set of work will form a patch series; but one advantage of *not* declaring a patch series, is that it allows others to work on it. e.g. It may be important for Sergei to put his name down on one of these further patches, if claiming a change under his/her name is an important issue :-).
85+ CJK characters for file names isn't too uncommon. I have non-english file names on my hard disk with say, 50+ characters.
If you insist on doing Chinese all the way, 127/3 = 42 is really a bit low for a descriptive string of any usage, but does anybody use non-english extended attributes on any OSes?
Hin-Tak
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2014-04-08 at 20:53 +0100, Hin-Tak Leung wrote:
[snip]
Quoted 5 lines
If you insist on doing Chinese all the way, 127/3 = 42 is really a bit low for a descriptive string of any usage, but does anybody use non-english extended attributes on any OSes?
If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+. And partial code changing in hfsplus_listxattr() is completely useless. Because getxattr() and setxattr() series of methods will be the primary source of error in the case of worst-case unicode to char conversion.
Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.
Thanks,
Vyacheslav Dubeyko.
Hi Vyacheslav,
Quoted 10 lines · Vyacheslav Dubeyko
On Tue, 2014-04-08 at 20:53 +0100, Hin-Tak Leung wrote:
[snip]Quoted 5 lines
If you insist on doing Chinese all the way, 127/3 = 42 is really a bit low for a descriptive string of any usage, but does anybody use non-english extended attributes on any OSes?
If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+.
It doesn't matter what anyone things. It matters to do what is right. And right is to allow anything that would work on OSX.
Quoted 1 line
And partial code changing in
hfsplus_listxattr()is completely useless.
No it is not at all useless. It is perfectly valid to run listxattr() without then running getxattr() on all the attributes! If an application cares about attributes foo and bar and there is also an attribute with a super long name then as things are at the moment, the listxattr() will fail with EIO error and the application will fail badly probably telling you your file system is corrupt. With the fix from Hin-Tak, the listxattr() will work fine and then the application will work fine because it will only getxattr() foo and bar. The difference being an application failing due to apparently corrupt file system or broken disk and an application working. How can you possibly state this fix is useless!?! Unbelievable...
Quoted 6 lines
Because
getxattr()andsetxattr()series of methods will be the primary source of error in the case of worst-case unicode to char conversion.Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.
Yes but I don't think you have given a good reason why it would be worth to create a kmem cache for such temporary buffers and IMO it makes no sense to use them - kmalloc and friends are a better choice in this use case. Please remember that kmalloc is effectively using a kmem cache - just a set of generic ones (depending on size of allocation - just look at /proc/slabinfo some time) instead of a specialised one so there is no benefit in using a dedicated kmem cache unless you are going to have lots of those buffers around which you are not as they only live for the duration of the get/set/list function call.
Best regards,
Anton
Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge J.J. Thomson Avenue, Cambridge, CB3 0RB, UK
Hi Anton,
Quoted 6 lines · Anton Altaparmakov
Quoted 3 lines
If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+.
It doesn't matter what anyone things. It matters to do what is right. And right is to allow anything that would work on OSX.
Yes, right things should be right. :)
I misunderstand where you've found in my sentence differences with your statement "to allow anything that would work on OSX". So, I see only useless objection here. Sorry.
Quoted 4 lines
Quoted 1 line
And partial code changing in
hfsplus_listxattr()is completely useless.No it is not at all useless. It is perfectly valid to run
listxattr()without then runninggetxattr()on all the attributes! If an application cares about attributes foo and bar and there is also an attribute with a super long name then as things are at the moment, thelistxattr()will fail with EIO error and the application will fail badly probably telling you your file system is corrupt. With the fix from Hin-Tak, thelistxattr()will work fine and then the application will work fine because it will onlygetxattr()foo and bar. The difference being an application failing due to apparently corrupt file system or broken disk and an application working. How can you possibly state this fix is useless!?! Unbelievable...
Yes, it's unbelievable that another guy has another opinion. :) I have another vision here. Sorry. But I have right to think in other way and to share my opinion freely. And I don't think that your vision is single and only way.
Quoted 9 lines
Quoted 6 lines
Because
getxattr()andsetxattr()series of methods will be the primary source of error in the case of worst-case unicode to char conversion.Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.
Yes but I don't think you have given a good reason why it would be worth to create a kmem cache for such temporary buffers and IMO it makes no sense to use them - kmalloc and friends are a better choice in this use case. Please remember that kmalloc is effectively using a kmem cache - just a set of generic ones (depending on size of allocation - just look at
/proc/slabinfosome time) instead of a specialised one so there is no benefit in using a dedicated kmem cache unless you are going to have lots of those buffers around which you are not as they only live for the duration of the get/set/list function call.
I see similarity between xattr name buffers and inodes buffers. So, I've suggested to use kmem cache because of such similarity. As far as I can judge, you mean that xattr name buffers will live short time. And, as a result, kmalloc will be much better choice. Do you mean this? Am I correct? Otherwise, it makes sense to use simple kmalloc and for inode buffers.
Thanks,
Vyacheslav Dubeyko.
Hi Vyacheslav,
Quoted 6 lines · Vyacheslav Dubeyko
Quoted 4 lines · Anton Altaparmakov
Quoted 2 lines
If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+.
It doesn't matter what anyone things. It matters to do what is right. And right is to allow anything that would work on OSX.
Quoted 7 lines
Yes, right things should be right. :)
I misunderstand where you've found in my sentence differences with your statement "to allow anything that would work on OSX". So, I see only useless objection here. Sorry.
From what I understood your sentence to mean (and I apologize if I misunderstood you) you said that "unless you have example or knowledge of someone using non-english extended attributes then there is no need to fix this issue on HFS+". And I think that is very wrong. Just because we don't have an example is not a good reason not to fix something. If it works on OSX which it does, then it should work on Linux, too. No matter whether anyone can provide an example of someone using it or not. It should be fixed simply because it is wrong and that is something that is obviously the case. The Linux kernel is not a business where the mentality of "if there is no customer bug report stop using company time on fixing it" is common place. Here we strive to always be correct. And what you sa
id (from what I understood you to mean anyway) was exactly the same as that business mentality of "if no example == customer then don't fix it"...
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Because
getxattr()andsetxattr()series of methods will be the primary source of error in the case of worst-case unicode to char conversion.Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.
Quoted 8 lines
Quoted 1 line
Yes but I don't think you have given a good reason why it would be worth to create a kmem cache for such temporary buffers and IMO it makes no sense to use them - kmalloc and friends are a better choice in this use case. Please remember that kmalloc is effectively using a kmem cache - just a set of generic ones (depending on size of allocation - just look at
/proc/slabinfosome time) instead of a specialised one so there is no benefit in using a dedicated kmem cache unless you are going to have lots of those buffers around which you are not as they only live for the duration of the get/set/list function call.I see similarity between xattr name buffers and inodes buffers. So, I've suggested to use kmem cache because of such similarity. As far as I can judge, you mean that xattr name buffers will live short time. And, as a result, kmalloc will be much better choice. Do you mean this? Am I correct? Otherwise, it makes sense to use simple kmalloc and for inode buffers.
There is no similarity between inode buffers and xattr name buffers other than the fact that they are both buffers!
xattr name buffer is currently allocated on the stack thus the buffer only exists for the duration of the function and is then thrown away. Thus at any one point in time there will be usually zero such buffers in memory and even if several processes simultaneously decide to do xattr related operations you would still at any one point in time have a maximum of "number of processes" such buffers allocated in memory.
inode buffers are very long lived buffers, on any typical system there are tens of thousands of them in memory at any one time and they stay in memory for a long time some even the entire time the kernel is running. On a busy server the number of inode buffer will go into the millions all in memory at same time.
Just the overhead of allocating and maintaining a kmem cache just for those almost never used xattr name buffers will be much larger in terms of memory and CPU use than the buffers you will allocate themselves. Thus it makes no sense to use kmem cache - just using kmalloc will use the appropriate, generic kmalloc kmem cache.
How can you possibly be comparing these two things and saying they are similar?!? You are comparing apples to oranges...
Quoted 1 line
From a completely idle, small server using ext3 file system for / which is 56GiB in size, looking at
/proc/slabinfowe can see the ext3 inode kmem cache and the generic 1024-byte kmem cache for kmalloc:
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext3_inode_cache 15105 15105 816 5 1 : tunables 54 27 8 : slabdata 3021 3021 size-1024 907 968 1024 4 1 : tunables 54 27 8 : slabdata 242 242 0
So there are 15105 cached "struct inode" on the root file system and at present there are 907 1024-byte sized kmalloc() buffers in memory.
Thus allocating a handful or even 10s or 100s more of 1024-byte size kmalloc() buffers is hardly going to make any difference to anything and fits nicely in what the size-1024 kmem cache is for.
The amount of wasted ram is absolutely minimal as the number of xattr name buffers is so small.
I think you really do not understand how memory allocators work... )-:
Best regards,
Anton
Quoted 2 lines
Thanks,
Vyacheslav Dubeyko.
Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge J.J. Thomson Avenue, Cambridge, CB3 0RB, UK