Skip to contentSkip to conversation

hfsplus: fixes worst-case unicode to char conversion of file names and attributes

PATCH Hin-Tak Leung 3 participants 5 messages Apr 8, 2014 – Apr 10, 2014 VFS

lore
Share & tools
lore
#0 lore raw

------------------------------

Quoted 14 lines · BST Vyacheslav Dubeyko

On Apr 8, 2014, at 9:09 PM, Anton Altaparmakov wrote:

[snip]

Quoted 8 lines

Unfortunately, it doesn't make sense to have fix only in hfsplus_listxattr(). It fixes nothing for xattr names. Of course, you can mention name of the patch. :) But such partial fix doesn't make sense for me. It will be more clear way to have one patch for file names and another patch for xattr names.

You are entitled to your opinion but I agree with Hin-Tak.  This makes perfect sense as it fixes the use of a function which can otherwise cause an erroneous EIO error to be returned from listxattr().  This in itself is a worthwhile fix.  Your idea of having to fix everything or nothing at all is frankly quite silly and would mean that hardly anything ever gets fixed...  Fixing small problems individually is just fine, especially when they are such obvious ones.

Quoted 6 lines

I have another opinion. :) But, of course, it is possible to see on things in different ways. I don't think that topic of splitting fixes between patches is really important. But my suggestion has more logical basis.

But, anyway, I suggest to use kmem cache for the case of xattr names instead of kmalloc.

This patch fix usages of hfsplus_uni2asc(). There are only two.

I think we all agree that there is a lot more work to be done in the attribute side to make sure that getting and setting long non-english attributes to work. And there will be a series of patches to it. The discussion is really on how this is split.

There are two uses of HFSPLUS_MAX_STRLEN concern asc2uni(), and are correct where they are. There is one other use in

super.c:319: buf->f_namelen = HFSPLUS_MAX_STRLEN;

That may need some thinking. It is probably correct.

For HFSPLUS_ATTR_MAX_STRLEN, besides the one I changed, there are 9 other uses as size of buffers (and a similar number in checks for length over-run, etc).

As I wrote earlier, some of these are setters, rather than getters. Setters and getters are different. The setters probably should use a large enough buffer according to userland, (ie. not a file-system specific one), and let the lower-layer of the file system returns an error when one is trying to set a too-long attribute. The getters needs to cope with what listxttr() returns.

So I see there should be at least two forth-coming patches related to attributes. One can bundle the attribute part of the current patch to the latter; but then that's assuming that the getter part of these further patches is logically one. It may be complex enough to require splitting up. We have 9 buffers, divisible at least now into two groups, getters and setters - it might need to be divided further. There are OS-prefixes complications along the way, so we may end up having to do 3-4 additional patches.

At this point it is not obvious that half of the current patch should be deferred, and where/what it should defer to. So I am thinking just "commit early, commit often".

Retrospectively - when we look back on it afterwards, in a few month's time -, the whole set of work will form a patch series; but one advantage of *not* declaring a patch series, is that it allows others to work on it. e.g. It may be important for Sergei to put his name down on one of these further patches, if claiming a change under his/her name is an important issue :-).

85+ CJK characters for file names isn't too uncommon. I have non-english file names on my hard disk with say, 50+ characters.

If you insist on doing Chinese all the way, 127/3 = 42 is really a bit low for a descriptive string of any usage, but does anybody use non-english extended attributes on any OSes?

Hin-Tak

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw

On Tue, 2014-04-08 at 20:53 +0100, Hin-Tak Leung wrote:

[snip]

Quoted 5 lines

If you insist on doing Chinese all the way, 127/3 = 42 is really a bit low for a descriptive string of any usage, but does anybody use non-english extended attributes on any OSes?

If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+. And partial code changing in hfsplus_listxattr() is completely useless. Because getxattr() and setxattr() series of methods will be the primary source of error in the case of worst-case unicode to char conversion.

Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.

Thanks,
Vyacheslav Dubeyko.

#2 lore raw

Hi Vyacheslav,

Quoted 10 lines · Vyacheslav Dubeyko

On Tue, 2014-04-08 at 20:53 +0100, Hin-Tak Leung wrote:
[snip]

Quoted 5 lines

If you insist on doing Chinese all the way, 127/3 = 42 is really a bit low for a descriptive string of any usage, but does anybody use non-english extended attributes on any OSes?

If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+.

It doesn't matter what anyone things. It matters to do what is right. And right is to allow anything that would work on OSX.

Quoted 1 line

And partial code changing in hfsplus_listxattr() is completely useless.

No it is not at all useless. It is perfectly valid to run listxattr() without then running getxattr() on all the attributes! If an application cares about attributes foo and bar and there is also an attribute with a super long name then as things are at the moment, the listxattr() will fail with EIO error and the application will fail badly probably telling you your file system is corrupt. With the fix from Hin-Tak, the listxattr() will work fine and then the application will work fine because it will only getxattr() foo and bar. The difference being an application failing due to apparently corrupt file system or broken disk and an application working. How can you possibly state this fix is useless!?! Unbelievable...

Quoted 6 lines

Because getxattr() and setxattr() series of methods will be the primary source of error in the case of worst-case unicode to char conversion.

Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.

Yes but I don't think you have given a good reason why it would be worth to create a kmem cache for such temporary buffers and IMO it makes no sense to use them - kmalloc and friends are a better choice in this use case. Please remember that kmalloc is effectively using a kmem cache - just a set of generic ones (depending on size of allocation - just look at /proc/slabinfo some time) instead of a specialised one so there is no benefit in using a dedicated kmem cache unless you are going to have lots of those buffers around which you are not as they only live for the duration of the get/set/list function call.

Best regards,

Anton

Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge
J.J. Thomson Avenue, Cambridge, CB3 0RB, UK
#3 lore raw

Hi Anton,

Quoted 6 lines · Anton Altaparmakov
Quoted 3 lines

If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+.

It doesn't matter what anyone things. It matters to do what is right. And right is to allow anything that would work on OSX.

Yes, right things should be right. :)

I misunderstand where you've found in my sentence differences with your statement "to allow anything that would work on OSX". So, I see only useless objection here. Sorry.

Quoted 4 lines
Quoted 1 line

And partial code changing in hfsplus_listxattr() is completely useless.

No it is not at all useless. It is perfectly valid to run listxattr() without then running getxattr() on all the attributes! If an application cares about attributes foo and bar and there is also an attribute with a super long name then as things are at the moment, the listxattr() will fail with EIO error and the application will fail badly probably telling you your file system is corrupt. With the fix from Hin-Tak, the listxattr() will work fine and then the application will work fine because it will only getxattr() foo and bar. The difference being an application failing due to apparently corrupt file system or broken disk and an application working. How can you possibly state this fix is useless!?! Unbelievable...

Yes, it's unbelievable that another guy has another opinion. :) I have another vision here. Sorry. But I have right to think in other way and to share my opinion freely. And I don't think that your vision is single and only way.

Quoted 9 lines
Quoted 6 lines

Because getxattr() and setxattr() series of methods will be the primary source of error in the case of worst-case unicode to char conversion.

Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.

Yes but I don't think you have given a good reason why it would be worth to create a kmem cache for such temporary buffers and IMO it makes no sense to use them - kmalloc and friends are a better choice in this use case. Please remember that kmalloc is effectively using a kmem cache - just a set of generic ones (depending on size of allocation - just look at /proc/slabinfo some time) instead of a specialised one so there is no benefit in using a dedicated kmem cache unless you are going to have lots of those buffers around which you are not as they only live for the duration of the get/set/list function call.

I see similarity between xattr name buffers and inodes buffers. So, I've suggested to use kmem cache because of such similarity. As far as I can judge, you mean that xattr name buffers will live short time. And, as a result, kmalloc will be much better choice. Do you mean this? Am I correct? Otherwise, it makes sense to use simple kmalloc and for inode buffers.

Thanks,
Vyacheslav Dubeyko.

#4 lore raw

Hi Vyacheslav,

Quoted 6 lines · Vyacheslav Dubeyko
Quoted 4 lines · Anton Altaparmakov
Quoted 2 lines

If you think that nobody use non-english extended attributes then it doesn't make sense to make any fixes in xattr subsystem of HFS+.

It doesn't matter what anyone things. It matters to do what is right. And right is to allow anything that would work on OSX.

Quoted 7 lines

Yes, right things should be right. :)

I misunderstand where you've found in my sentence differences with your statement "to allow anything that would work on OSX". So, I see only useless objection here. Sorry.

From what I understood your sentence to mean (and I apologize if I misunderstood you) you said that "unless you have example or knowledge of someone using non-english extended attributes then there is no need to fix this issue on HFS+". And I think that is very wrong. Just because we don't have an example is not a good reason not to fix something. If it works on OSX which it does, then it should work on Linux, too. No matter whether anyone can provide an example of someone using it or not. It should be fixed simply because it is wrong and that is something that is obviously the case. The Linux kernel is not a business where the mentality of "if there is no customer bug report stop using company time on fixing it" is common place. Here we strive to always be correct. And what you sa

id (from what I understood you to mean anyway) was exactly the same as that business mentality of "if no example == customer then don't fix it"...

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Because getxattr() and setxattr() series of methods will be the primary source of error in the case of worst-case unicode to char conversion.

Otherwise, you can use any splitting of fixes between patches. In my last e-mail I suggested to use kmem cache approach for xattr names instead of kmalloc only.

Quoted 8 lines
Quoted 1 line

Yes but I don't think you have given a good reason why it would be worth to create a kmem cache for such temporary buffers and IMO it makes no sense to use them - kmalloc and friends are a better choice in this use case. Please remember that kmalloc is effectively using a kmem cache - just a set of generic ones (depending on size of allocation - just look at /proc/slabinfo some time) instead of a specialised one so there is no benefit in using a dedicated kmem cache unless you are going to have lots of those buffers around which you are not as they only live for the duration of the get/set/list function call.

I see similarity between xattr name buffers and inodes buffers. So, I've suggested to use kmem cache because of such similarity. As far as I can judge, you mean that xattr name buffers will live short time. And, as a result, kmalloc will be much better choice. Do you mean this? Am I correct? Otherwise, it makes sense to use simple kmalloc and for inode buffers.

There is no similarity between inode buffers and xattr name buffers other than the fact that they are both buffers!

xattr name buffer is currently allocated on the stack thus the buffer only exists for the duration of the function and is then thrown away. Thus at any one point in time there will be usually zero such buffers in memory and even if several processes simultaneously decide to do xattr related operations you would still at any one point in time have a maximum of "number of processes" such buffers allocated in memory.

inode buffers are very long lived buffers, on any typical system there are tens of thousands of them in memory at any one time and they stay in memory for a long time some even the entire time the kernel is running. On a busy server the number of inode buffer will go into the millions all in memory at same time.

Just the overhead of allocating and maintaining a kmem cache just for those almost never used xattr name buffers will be much larger in terms of memory and CPU use than the buffers you will allocate themselves. Thus it makes no sense to use kmem cache - just using kmalloc will use the appropriate, generic kmalloc kmem cache.

How can you possibly be comparing these two things and saying they are similar?!? You are comparing apples to oranges...

Quoted 1 line

From a completely idle, small server using ext3 file system for / which is 56GiB in size, looking at /proc/slabinfo we can see the ext3 inode kmem cache and the generic 1024-byte kmem cache for kmalloc:

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext3_inode_cache   15105  15105    816    5    1 : tunables   54   27    8 : slabdata   3021   3021  
size-1024            907    968   1024    4    1 : tunables   54   27    8 : slabdata    242    242      0

So there are 15105 cached "struct inode" on the root file system and at present there are 907 1024-byte sized kmalloc() buffers in memory.

Thus allocating a handful or even 10s or 100s more of 1024-byte size kmalloc() buffers is hardly going to make any difference to anything and fits nicely in what the size-1024 kmem cache is for.

The amount of wasted ram is absolutely minimal as the number of xattr name buffers is so small.

I think you really do not understand how memory allocators work... )-:

Best regards,

Anton

Quoted 2 lines

Thanks,
Vyacheslav Dubeyko.

Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge
J.J. Thomson Avenue, Cambridge, CB3 0RB, UK