Skip to contentSkip to conversation

Linux UDF support

DISCUSSION Pali Rohár 3 participants 6 messages Aug 24, 2014 – Aug 26, 2014

lore
Share & tools
lore
#0 lore raw

Hi,

I would like to know what is state of linux UDF driver. It is experimental or is now suitable for storing data?

According to wikipedia [1] UDF has open specification format and can be used also for HDDs (not only optical discs).

In OS support table is written that all major and other minor OSs support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. Are there any disadvantages for using UDF on e.g USB flash disk? (when I want read/write support on Linux, Windows 7 and Mac OS X)

Because lot of manuals say that FAT32 (or NTFS) is only one solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only up to UDF version 2.01. Is this restriction still valid?

What will happen if I try to mount FS with UDF version 2.60 in R/W mode on linux? It will fallback to R/O mode? Or newly written files will be in previous (2.01) versions?

And last question: Is there some fsck tool for UDF? Or at least tool which print if FS is in inconsistent state?

[1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

Signature · 2 lines
Pali Rohár
[email protected]
#1 lore raw
Quoted 5 lines · Pali Rohár

Hi,

I would like to know what is state of linux UDF driver. It is experimental or is now suitable for storing data?

I know that read support works for every version I have tested, but I've only tested it reading data from DVD's and Blu-Ray discs, so I don't know how well it works for other purposes.

Quoted 9 lines

According to wikipedia [1] UDF has open specification format and can be used also for HDDs (not only optical discs).

In OS support table is written that all major and other minor OSs support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. Are there any disadvantages for using UDF on e.g USB flash disk? (when I want read/write support on Linux, Windows 7 and Mac OS X)

If you are going to go that way, make sure to use the Spared Build, as otherwise you will run in to the same media wear-out issues that NTFS and FAT have. Also, keep in mind that pre-Vista Windows and pre-10.4 OSX don't have very good support for the newer formats.

Quoted 5 lines

Because lot of manuals say that FAT32 (or NTFS) is only one solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only up to UDF version 2.01. Is this restriction still valid?

I do know that we support reading UDF 2.60 (I've used linux to read Blu-Ray discs), but I have no idea about write support for versions above 2.01.

Quoted 6 lines

What will happen if I try to mount FS with UDF version 2.60 in R/W mode on linux? It will fallback to R/O mode? Or newly written files will be in previous (2.01) versions?

And last question: Is there some fsck tool for UDF? Or at least tool which print if FS is in inconsistent state?

Most Linux distributions have a package called udftools, the upstream URL given by portage is http://sf.net/projects/linux-udf/

Quoted 3 lines

[1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

#2 lore raw

Hi,

Quoted 11 lines · On Monday 25 August 2014 14:45:13 Austin S Hemmelgarn wrote:
Quoted 4 lines · Pali Rohár

Hi,

I would like to know what is state of linux UDF driver. It is experimental or is now suitable for storing data?

I know that read support works for every version I have tested, but I've only tested it reading data from DVD's and Blu-Ray discs, so I don't know how well it works for other purposes.

Ok. I'm thinking about using UDF on HDD and usb flash disks (not on optical medium). And here I need write support too.

Quoted 12 lines
Quoted 11 lines

According to wikipedia [1] UDF has open specification format and can be used also for HDDs (not only optical discs).

In OS support table is written that all major and other minor OSs support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. Are there any disadvantages for using UDF on e.g USB flash disk? (when I want read/write support on Linux, Windows 7 and Mac OS X)

Quoted 6 lines

If you are going to go that way, make sure to use the Spared Build, as otherwise you will run in to the same media wear-out issues that NTFS and FAT have. Also, keep in mind that pre-Vista Windows and pre-10.4 OSX don't have very good support for the newer formats.

What is Spared Build? And how to use it?

Problem with NTFS is that linux driver has write support marked as experimental. FAT has problems with big files and for exFAT there is no driver in linux kernel yet...

Quoted 15 lines
Quoted 6 lines

Because lot of manuals say that FAT32 (or NTFS) is only one solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only up to UDF version 2.01. Is this restriction still valid?

I do know that we support reading UDF 2.60 (I've used linux to read Blu-Ray discs), but I have no idea about write support for versions above 2.01.

Quoted 4 lines

What will happen if I try to mount FS with UDF version 2.60 in R/W mode on linux? It will fallback to R/O mode? Or newly written files will be in previous (2.01) versions?

Quoted 7 lines
Quoted 2 lines

And last question: Is there some fsck tool for UDF? Or at least tool which print if FS is in inconsistent state?

Most Linux distributions have a package called udftools, the upstream URL given by portage is
http://sf.net/projects/linux-udf/

That project does not have udf fsck tool :-(

Quoted 1 line
Quoted 1 line

[1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

Ok, I will wait for response from maintainer Jan, he probably would know more...

Signature · 2 lines
Pali Rohár
[email protected]
#3 lore raw
Quoted 30 lines · Pali Rohár

Hi,

Quoted 11 lines · On Monday 25 August 2014 14:45:13 Austin S Hemmelgarn wrote:
Quoted 4 lines · Pali Rohár

Hi,

I would like to know what is state of linux UDF driver. It is experimental or is now suitable for storing data?

I know that read support works for every version I have tested, but I've only tested it reading data from DVD's and Blu-Ray discs, so I don't know how well it works for other purposes.

Ok. I'm thinking about using UDF on HDD and usb flash disks (not on optical medium). And here I need write support too.

Quoted 12 lines
Quoted 11 lines

According to wikipedia [1] UDF has open specification format and can be used also for HDDs (not only optical discs).

In OS support table is written that all major and other minor OSs support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. Are there any disadvantages for using UDF on e.g USB flash disk? (when I want read/write support on Linux, Windows 7 and Mac OS X)

Quoted 8 lines
Quoted 6 lines

If you are going to go that way, make sure to use the Spared Build, as otherwise you will run in to the same media wear-out issues that NTFS and FAT have. Also, keep in mind that pre-Vista Windows and pre-10.4 OSX don't have very good support for the newer formats.

What is Spared Build? And how to use it?

Sparse Build is one of the extensions that was added in 1.50. It provides a table indicating sectors that have worn out and therefore should be left unused. The general idea is that rewritable media is almost always limited in the number of writes it can handle to a given location, and when you exceed that number, you get data corruption at that location. This is usually most noticeable on flash media, but it happens on {CD,DVD,BD}-RW discs and hard drives as well. FAT has no provisions for handling this, and NTFS and ext4 just return the corrupted data. UDF however, has sufficient error correction ability (because it was designed with optical media in mind) that it can usually detect the corruption, recover the corrupted data, and then if you are using the Sparse Build format, mark that block as unusable, and write the new data out elsewhere.

Quoted 5 lines

Problem with NTFS is that linux driver has write support marked as experimental. FAT has problems with big files and for exFAT there is no driver in linux kernel yet...

For NTFS, you should be using NTFS-3G, not the kernel driver, and I personally wouldn't trust the OS X driver for NTFS for anything beyond read-only usage (I only barely trust NTFS-3G as it is, and have about as much trust in the Linux kernel driver for it as I would in somebody trying to convince me that arsenic is good for you). And for exFAT there is FUSE module. I haven't tested either for much other than read-only scenarios, but I can confirm that they are great for data recovery.

Quoted 18 lines
Quoted 18 lines
Quoted 6 lines

Because lot of manuals say that FAT32 (or NTFS) is only one solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only up to UDF version 2.01. Is this restriction still valid?

I do know that we support reading UDF 2.60 (I've used linux to read Blu-Ray discs), but I have no idea about write support for versions above 2.01.

Quoted 6 lines

What will happen if I try to mount FS with UDF version 2.60 in R/W mode on linux? It will fallback to R/O mode? Or newly written files will be in previous (2.01) versions?

And last question: Is there some fsck tool for UDF? Or at least tool which print if FS is in inconsistent state?

Quoted 6 lines
Quoted 4 lines

Most Linux distributions have a package called udftools, the upstream URL given by portage is
http://sf.net/projects/linux-udf/

That project does not have udf fsck tool :-(

IIRC, there are a lot of issues that UDF will recover from gracefully. It has ridiculously good error correction abilities, and the newer versions even have the option of duplicate copies of filesystem metadata to provide even more resilience.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

[1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

Ok, I will wait for response from maintainer Jan, he probably would know more...

#4 lore raw

Hi,

Quoted 39 lines · On Monday 25 August 2014 16:05:27 Austin S Hemmelgarn wrote:
Quoted 30 lines · Pali Rohár

Hi,

Quoted 10 lines · On Monday 25 August 2014 14:45:13 Austin S Hemmelgarn wrote:
Quoted 4 lines · Pali Rohár

Hi,

I would like to know what is state of linux UDF driver. It is experimental or is now suitable for storing data?

I know that read support works for every version I have tested, but I've only tested it reading data from DVD's and Blu-Ray discs, so I don't know how well it works for other purposes.

Ok. I'm thinking about using UDF on HDD and usb flash disks (not on optical medium). And here I need write support too.

Quoted 13 lines
Quoted 12 lines

According to wikipedia [1] UDF has open specification format and can be used also for HDDs (not only optical discs).

In OS support table is written that all major and other minor OSs support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. Are there any disadvantages for using UDF on e.g USB flash disk? (when I want read/write support on Linux, Windows 7 and Mac OS X)

Quoted 7 lines
Quoted 5 lines

If you are going to go that way, make sure to use the Spared Build, as otherwise you will run in to the same media wear-out issues that NTFS and FAT have. Also, keep in mind that pre-Vista Windows and pre-10.4 OSX don't have very good support for the newer formats.

What is Spared Build? And how to use it?

Quoted 16 lines

Sparse Build is one of the extensions that was added in 1.50. It provides a table indicating sectors that have worn out and therefore should be left unused. The general idea is that rewritable media is almost always limited in the number of writes it can handle to a given location, and when you exceed that number, you get data corruption at that location. This is usually most noticeable on flash media, but it happens on {CD,DVD,BD}-RW discs and hard drives as well. FAT has no provisions for handling this, and NTFS and ext4 just return the corrupted data. UDF however, has sufficient error correction ability (because it was designed with optical media in mind) that it can usually detect the corruption, recover the corrupted data, and then if you are using the Sparse Build format, mark that block as unusable, and write the new data out elsewhere.

Looks good. But I think that when I reformat disk I will lost all these data about detected corruption... Or will duplicate mkudffs calls preserve these information about HW errors?

So if I format block device (hdd) with mkudffs and rev 0x0201 it is automatically enabled? Or is something more needed?

Quoted 4 lines
Quoted 3 lines

Problem with NTFS is that linux driver has write support marked as experimental. FAT has problems with big files and for exFAT there is no driver in linux kernel yet...

Quoted 10 lines

For NTFS, you should be using NTFS-3G, not the kernel driver, and I personally wouldn't trust the OS X driver for NTFS for anything beyond read-only usage (I only barely trust NTFS-3G as it is, and have about as much trust in the Linux kernel driver for it as I would in somebody trying to convince me that arsenic is good for you). And for exFAT there is FUSE module. I haven't tested either for much other than read-only scenarios, but I can confirm that they are great for data recovery.

I'm not sure if using fuse for anything other then experiments is reasonable. And speed of fuse (ntfs-3g or exfat) is also horrible. And now I think that only usage is read-only data recovery...

So still would like to hear what is state of udf read/write support in kernel (for usb/flash/hdd disks).

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 6 lines

Because lot of manuals say that FAT32 (or NTFS) is only one solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only up to UDF version 2.01. Is this restriction still valid?

I do know that we support reading UDF 2.60 (I've used linux to read Blu-Ray discs), but I have no idea about write support for versions above 2.01.

Quoted 7 lines

What will happen if I try to mount FS with UDF version 2.60 in R/W mode on linux? It will fallback to R/O mode? Or newly written files will be in previous (2.01) versions?

And last question: Is there some fsck tool for UDF? Or at least tool which print if FS is in inconsistent state?

Most Linux distributions have a package called udftools, the upstream URL given by portage is
http://sf.net/projects/linux-udf/

Quoted 8 lines
Quoted 1 line

That project does not have udf fsck tool :-(

IIRC, there are a lot of issues that UDF will recover from gracefully. It has ridiculously good error correction abilities, and the newer versions even have the option of duplicate copies of filesystem metadata to provide even more resilience.

But this depends on driver implementation, rigth? Is linux kernel driver able to recovery and fix (on the fly) errors at mount time (like ubifs or ext4)? Somebody said that ubifs does not need fsck because kernel driver doing similar job of recovery at mount time... It is same for udf implementation in linux?

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

[1] - http://en.wikipedia.org/wiki/Universal_Disk_Format

Ok, I will wait for response from maintainer Jan, he probably would know more...

Signature · 2 lines
Pali Rohár
[email protected]
#5 lore raw
In reply to Pali Rohár

Hello,

Quoted 2 lines · Pali Rohár

I would like to know what is state of linux UDF driver. It is experimental or is now suitable for storing data?

The kernel driver is quite stable. I would happily use it for USB sticks
etc. I would be reluctant to use it for say /home because we don't have any reasonable repair tool for the filesystem (there are format checkers out there but they mostly tell you there is a problem, however there's not a decent tool which would fix the filesystem AFAIK).

Quoted 2 lines

According to wikipedia [1] UDF has open specification format and can be used also for HDDs (not only optical discs).

Correct.

Quoted 6 lines

In OS support table is written that all major and other minor OSs support UDF FS (without needs for additional programs).

So it looks like UDF is good candidate for multi OS filesystem. Are there any disadvantages for using UDF on e.g USB flash disk? (when I want read/write support on Linux, Windows 7 and Mac OS X)

Yes, it is a good candidate for exchange between different OSs. I know
people who use it as such.

Quoted 5 lines

Because lot of manuals say that FAT32 (or NTFS) is only one solution for using USB flash disk on more OS.

On wikipedia there is one note about linux: Write support is only up to UDF version 2.01. Is this restriction still valid?

That is correct.

Quoted 3 lines

What will happen if I try to mount FS with UDF version 2.60 in R/W mode on linux? It will fallback to R/O mode? Or newly written files will be in previous (2.01) versions?

The kernel refuses to mount the filesystem read-write if its revision is

Quoted 1 line

2.01. But frankly for interchange between OSs you don't want to use

advanced features anyway (lower chance fs will be supported) and there's not much benefit since they are mostly aimed at optical media.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR