Skip to contentSkip to conversation

PROBLEM: hfsplus filesystem allows opendir() on plain files

DISCUSSION Damian Stewart 5 participants 11 messages Dec 2, 2008 – Dec 6, 2008 VFS

lore
Share & tools
lore
#0 lore raw

1. hfsplus filesystem allows opendir() on plain files.

2. When opendir() is called on a plain file stored in an hfsplus filesystem, opendir() fails to return 0 and set errno to ENOTDIR.

3. filesystem, hfsplus

4. Linux version 2.6.27-7-generic (buildd@palmer) (gcc version 4.3.2 (Ubuntu 4.3.2-1ubuntu11) ) #1 SMP Tue Nov 4 19:33:20 UTC 2008

5. unknown

6. n/a

7. Try the following program on files and directories first under a mountpoint for an ext3 partition then under a mountpoint for an hfsplus partition.

///////////////////////////////////
#include <dirent.h>
#include <stdio.h>
#include <errno.h>

// usage: opendir_test <file>|<directory>

int main( int argc, char** argv )

{
    DIR* result = opendir( argv[1] );
    printf("ENOTDIR is 0x%x\n", ENOTDIR );
    printf("opendir '%s' returned 0x%x, errno 0x%x\n", argv[1], result, 
errno );
    return 0;
}

//////////////////////////////////

I get the following output:

damian@damian-laptop:~/code/opendir$ mkdir test_directory
damian@damian-laptop:~/code/opendir$ echo hello > test_file.txt
damian@damian-laptop:~/code/opendir$ ./opendir_test test_directory
ENOTDIR is 0x14
opendir 'test_directory' returned 0x88f6008, errno 0x0
damian@damian-laptop:~/code/opendir$ ./opendir_test test_file.txt
ENOTDIR is 0x14
opendir 'test_file.txt' returned 0x0, errno 0x14
# /osx is an HFSPLUS mountpoint
damian@damian-laptop:~/code/opendir$ cd /media/osx
# mkdir test_directory && echo hello > test_file.txt performed in OSX
damian@damian-laptop:/media/osx$ ./opendir_test test_directory
ENOTDIR is 0x14
opendir 'test_directory' returned 0x919d008, errno 0x0
# erroneous result follows
damian@damian-laptop:/media/osx$ ./opendir_test test_file.txt
ENOTDIR is 0x14
opendir 'test_file.txt' returned 0x9a23008, errno 0x0
damian@damian-laptop:/media/osx$

8. If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.

Linux damian-laptop 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:20 UTC 2008 i686 GNU/Linux

Gnu C                  4.3.2-1ubuntu11)
Gnu make               3.81
binutils               Binutils
util-linux             2.14
mount                  support
module-init-tools      3.3-pre11
e2fsprogs              1.41.3
reiserfsprogs          3.6.19
pcmciautils            014
Linux C Library        2.8.90
Dynamic linker (ldd)   2.8.90
Procps                 3.2.7
Net-tools              1.60
Kbd                    1.14.1
Sh-utils               6.10
udev                   124
wireless-tools         29
Modules Loaded         ipv6 aes_i586 aes_generic nls_utf8 hfsplus i915 drm af_packet binfmt_misc btusb sco bridge stp bnep rfcomm l2cap bluetooth ppdev acpi_cpufreq cpufreq_stats cpufreq_powersave cpufreq_conservative cpufreq_userspace cpufreq_ondemand freq_table container wmi pci_slot sbs sbshc iptable_filter ip_tables x_tables sbp2 parport_pc lp parport joydev snd_hda_intel arc4 snd_pcm_oss snd_mixer_oss iTCO_wdt ecb crypto_blkcipher iTCO_vendor_support snd_pcm ath9k evdev appleir pcspkr appletouch snd_seq_dummy mac80211 snd_seq_oss cfg80211 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq video snd_timer snd_seq_device output battery snd ac intel_agp button soundcore agpgart shpchp pci_hotplug snd_page_alloc ext3 jbd mbcache sd_mod crc_t10dif sr_mod cdrom sg ata_generic ata_piix usbhid hid pata_acpi ohci1394 libata ieee1394 scsi_mod ehci_hcd uhci_hcd dock usbcore sky2 thermal processor fan fbcon tileblit font bitblit softcursor fuse

thanks
d

Signature · 2 lines
damian stewart | skype: damiansnz | [email protected]
frey | live art with machines | http://www.frey.co.nz
#1 lore raw
Quoted 4 lines · Damian Stewart

1. hfsplus filesystem allows opendir() on plain files.

2. When opendir() is called on a plain file stored in an hfsplus filesystem, opendir() fails to return 0 and set errno to ENOTDIR.

Yes, this is a side effect of an intentional behavior. In osx, you can open "file/rsrc" to get the resource fork of "file". This behavior is emulated inside hfsplus on Linux, which means that to some degree every file acts like a directory. There isn't a readdir operation so it's not possible to list these, but it is possible to do a lookup on this specific name. The code path taken by opendir inside the kernel checks inode->i_op->lookup and returns ENOTDIR if this function pointer is not set.

We can't fix this without either eliminating this way to open the resource fork or making real changes to the generic Linux code. I'm not sure what would be the appropriate thing to do in this case. Someone on the mailing list is likely to have a strong opinion, so there are likely to be suggestions of one sort or another.

Maybe the check in the generic code to decide what is a "directory" should be changed? We could possibly check inode->i_fop->readdir.

Brad Boyer
[email protected]

#2 lore raw
Quoted 2 lines · Brad Boyer

Maybe the check in the generic code to decide what is a "directory" should be changed? We could possibly check inode->i_fop->readdir.

Hi Brad,

Thanks for your reply.

This issue came up because at least one major project (the VLC video player) is using 'opendir()' to determine whether or not a path is a file or directory. (They probably ought to be using 'stat()', but they're not, so this issue shows up.)

I've reported it on the VLC forums but only got very dismissive replies. The few developers who posted bluntly said it was a 'kernel bug' so they weren't willing to do anything about it; which is a shame, because it leaves those VLC users with videos on HFS+ drives unable to play their videos with the latest version of VLC.

Anyways, perhaps your suggestion is the right one, in this case.

Cheers
d

Signature · 2 lines
damian stewart | skype: damiansnz | [email protected]
frey | live art with machines | http://www.frey.co.nz
#3 lore raw
Quoted 4 lines · Damian Stewart

This issue came up because at least one major project (the VLC video player) is using 'opendir()' to determine whether or not a path is a file or directory. (They probably ought to be using 'stat()', but they're not, so this issue shows up.)

Unless they're already needing to open any directories they check, doing an opendir call to check a file type seems less than ideal. Hopefully they had a reason, but I know how that sort of thing can happen.

Quoted 5 lines

I've reported it on the VLC forums but only got very dismissive replies. The few developers who posted bluntly said it was a 'kernel bug' so they weren't willing to do anything about it; which is a shame, because it leaves those VLC users with videos on HFS+ drives unable to play their videos with the latest version of VLC.

Well, I actually tend to agree that it is less than ideal behavior on the part of the kernel in spite of thinking that they are doing something stupid. It really ought to work.

Quoted 1 line

Anyways, perhaps your suggestion is the right one, in this case.

We'll see what comes out of this.

Brad Boyer
[email protected]

#4 lore raw
In reply to Brad Boyer
Quoted 2 lines · Brad Boyer

We can't fix this without either eliminating this way to open the resource fork

Which sould have happened long ago. See the reiser4 discussion. You're breaking userspace semantic without a good reason.

#5 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Brad Boyer

We can't fix this without either eliminating this way to open the resource fork

Which sould have happened long ago. See the reiser4 discussion. You're breaking userspace semantic without a good reason.

I personally don't care about the feature myself. I'm not sure what good it does anyway, and I don't know of anything on Linux that uses it. However, I also wasn't the one that added it. My original version of hfsplus did not have any way to access them. Maybe it's best to just remove it and see who complains about it. I do want to point out that I don't maintain that code anymore and don't have time to do that work myself. As far as I know, hfsplus is officially unmaintained these days.

Just as a side note, hfs has the same feature. The two modules are basically the same code these days.

Brad Boyer
[email protected]

maintainer
#6 lore raw
Quoted 11 lines · Brad Boyer

I personally don't care about the feature myself. I'm not sure what good it does anyway, and I don't know of anything on Linux that uses it. However, I also wasn't the one that added it. My original version of hfsplus did not have any way to access them. Maybe it's best to just remove it and see who complains about it. I do want to point out that I don't maintain that code anymore and don't have time to do that work myself. As far as I know, hfsplus is officially unmaintained these days.

Just as a side note, hfs has the same feature. The two modules are basically the same code these days.

You know what... I thought that HFS+ did not support links; looking at the source, it appears that hardlinks *are* supported and that brings all the deadlocks into the game, same as we had in reiser4 case.

Which is to say, this stuff *must* go. In case of HFS it's obnoxious, but not immediately letal. In case of HFS+ it's an instant DoS.

#7 lore raw
Quoted 3 lines · Al Viro

You know what... I thought that HFS+ did not support links; looking at the source, it appears that hardlinks *are* supported and that brings all the deadlocks into the game, same as we had in reiser4 case.

Apple added hard links with osx. Back when I wrote hfsplus originally, there wasn't any such thing on HFS+ either. The support for the hack that added the appearance of hard links was added after I stopped maintaining the code. It's really just a matter of hiding the real file in a special place and putting stubs with special settings where all the links appear in the namespace. In all truth both HFS and HFS+ are inherently incompatible with the very concept of a hard link or having a file still exist after removal. It's just a horrible hack that makes it look like it works.

Quoted 2 lines

Which is to say, this stuff *must* go. In case of HFS it's obnoxious, but not immediately letal. In case of HFS+ it's an instant DoS.

Feel free. I disavowed any responsibility for this code years ago. And before anyone thinks of complaining, I'll point out that I wasn't the one that submitted hfsplus for inclusion in the kernel either.

Brad Boyer
[email protected]

maintainer
#8 lore raw
Quoted 9 lines · Brad Boyer

Apple added hard links with osx. Back when I wrote hfsplus originally, there wasn't any such thing on HFS+ either. The support for the hack that added the appearance of hard links was added after I stopped maintaining the code. It's really just a matter of hiding the real file in a special place and putting stubs with special settings where all the links appear in the namespace. In all truth both HFS and HFS+ are inherently incompatible with the very concept of a hard link or having a file still exist after removal. It's just a horrible hack that makes it look like it works.

Well... That's not too unreasonably way to do them, actually - just treat your hidden directory as real inode table. The tricky part is keeping i_ino stable through all that, but if it's done...

FWIW, fs/qnx4 uses a somewhat similar scheme, IIRC.

To make things workable you need
* hidden directory that would never be reachable by syscalls
* recognizable files that would contain references to names
in that directory
* lookup that would normally lead to such file getting inode
of object in hidden directory
* link(2) with target being *not* one of those would start with
moving target to hidden directory and creating such "link" file in its place. Then it would create a link file in source. That, BTW, would have interesting implications wrt locking - we'd need to modify target's parent, which can get sucky.
* unlink(2) with victim dentry not being a "link" file *and* being
busy would move the victim to hidden. Otherwise it'd kill the victim outright.
* final iput() of file that'd been taken to hidden does deletion
of directory entry in hidden if i_nlink is 0.

The interesting part is locking - you need to deal with lookup vs. eviction of link(2) target to hidden and with rename vs. the same thing.

Quoted 6 lines
Quoted 2 lines

Which is to say, this stuff *must* go. In case of HFS it's obnoxious, but not immediately letal. In case of HFS+ it's an instant DoS.

Feel free. I disavowed any responsibility for this code years ago. And before anyone thinks of complaining, I'll point out that I wasn't the one that submitted hfsplus for inclusion in the kernel either.

Not blaming you...

maintainer
#9 lore raw
Quoted 35 lines · Al Viro
Quoted 9 lines · Brad Boyer

Apple added hard links with osx. Back when I wrote hfsplus originally, there wasn't any such thing on HFS+ either. The support for the hack that added the appearance of hard links was added after I stopped maintaining the code. It's really just a matter of hiding the real file in a special place and putting stubs with special settings where all the links appear in the namespace. In all truth both HFS and HFS+ are inherently incompatible with the very concept of a hard link or having a file still exist after removal. It's just a horrible hack that makes it look like it works.

Well... That's not too unreasonably way to do them, actually - just treat your hidden directory as real inode table. The tricky part is keeping i_ino stable through all that, but if it's done...

FWIW, fs/qnx4 uses a somewhat similar scheme, IIRC.

To make things workable you need
* hidden directory that would never be reachable by syscalls
* recognizable files that would contain references to names
in that directory
* lookup that would normally lead to such file getting inode
of object in hidden directory
* link(2) with target being *not* one of those would start with
moving target to hidden directory and creating such "link" file in its place. Then it would create a link file in source. That, BTW, would have interesting implications wrt locking - we'd need to modify target's parent, which can get sucky.
* unlink(2) with victim dentry not being a "link" file *and* being
busy would move the victim to hidden. Otherwise it'd kill the victim outright.
* final iput() of file that'd been taken to hidden does deletion
of directory entry in hidden if i_nlink is 0.

Quoted 2 lines

The interesting part is locking - you need to deal with lookup vs. eviction of link(2) target to hidden and with rename vs. the same thing.

PS: that's for handling link(2) on non-directories with that kind of layout, *NOT* for file-cross-directory hybrids or links to directories.

#10 lore raw
In reply to Al Viro

Hi,

Quoted 6 lines · Al Viro

You know what... I thought that HFS+ did not support links; looking at the source, it appears that hardlinks *are* supported and that brings all the deadlocks into the game, same as we had in reiser4 case.

Which is to say, this stuff *must* go. In case of HFS it's obnoxious, but not immediately letal. In case of HFS+ it's an instant DoS.

At least on OS X 10.5 (Leopard) HFS+ not only supports hard links but also supports hard links on directories (though they are read-only)!

btw. That is how time machine works (the built in backup tool that backs up to a HFS+ file system and then each hour it creates a new directory tree by doing hard links to the previous one including the directories but the backups are read-only).

Supporting that on Linux as it is today is likely to be impossible without major work which no-one is prepared to do / accept...

Best regards,

Anton

Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/