Moving unmaintained filesystems to staging

hfsplus - Mac filesystem.

minix

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We have no maintainer for hfs, and no likely prospect of anyone stepping up soon to become hfs maintainer. I think it's irresponsible of us to present unmaintained code on an equal basis with filesystems under active maintenance like ext2.

I therefore propose we move the following filesystems which are explicitly listed as Orphaned to staging:

affs - Amiga filesystem.
efs - old SGI filesystem predating XFS, used on CDs for a while. hfs - Mac filesystem.
hfsplus - Mac filesystem.

I further propose we move the following filesystems which have no entry in MAINTAINERS to staging:

adfs - Acorn filesystem from the 1990s.
minix
qnx6

I had similar toughts some time ago while browsing the fs/ directory. Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

Quoted 10 lines

I had similar toughts some time ago while browsing the fs/ directory.
Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

Yeah, it's pretty sad how few commits some of these filesystems have had in recent years. One can argue that they're stable and don't need to be fixed because they aren't broken, but I find it hard to believe that any of them were better-implemented than ext2 which still sees regular bugfixes.

Am Sonntag, 17. Juni 2012 schrieb jdow:

Quoted 2 lines · Geert Uytterhoeven

Quoted 1 line

On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald

Quoted 13 lines · <[email protected]> wrote:

Quoted 13 lines

Quoted 13 lines

Am Sonntag, 17. Juni 2012 schrieb jdow:

| JXFS 64 bit file system
|
| With AmigaOS 4.x a new file system has been introduced called
| JXFS. It is a totally new 64 bit file system that supports
| partitions up to 16 TB in size. It is a modern journalling file
| system, which means that it reduces data loss if data writes to
| the disk are interrupted. It is the fastest and most reliable
| file system ever created for AmigaOS.

http://www.amigaos.net/content/1/features

Well I asked AmigaOS 4 developers about this issue as well. Lets

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

what they say about 2 TB limits.

16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?

block/partitions/amiga.c reads the block size from RigidDiskBlock.rdb_BlockBytes,
but after conversion to 512-byte blocks, all further calculations

Quoted 5 lines

Quoted 5 lines

done on "int", so it will overflow for disks larger than 2 TiB.

Note that in your profile-binary.img, the field is 0x200, i.e. 512 bytes per block,
so I'll have to get a deeper look into your RDB first...

[…]

Quoted 2 lines

Note that the work I did on the Linux RDB code eons ago took full cognizance of the physical and virtual block sizes.

[…]

Quoted 1 line

I've asked Martin for a digital copy of his RDBs and what he thinks

Quoted 2 lines

partition(s) should look like. I should also be told whether the disk is supposed to be solely Amiga OSs or not. I gather it's not.

Its all in the bug report. profile-binary.img should be it.

Its small so I attach it.

Meanwhile I try to get the currently supported maximum disk size out

the OS 4 developers. Maybe JXFS is playing tricks that other

not play or simply has a different implementation.

Thanks,

As long as there's someone who can be pestered with bugs, I don't see the need to push their baby out of the tree. I'm a bit more nervous about hfsplus; if it has users and no maintainer, those users are at risk.

Perhaps we could advertise on Craigslist or something ... maintainer wanted for LTR.

You probably put your stick into a cave with ancient sleeping dragons :)

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.18/scsi-queue&id=3109e5ae0311e937d49a5325134e50b742ac5f4a
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=861928f4e60e826cd8871c0c37f4b3d825b8d81d
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/pata_gayle.c?id=9ab27d1d35fda0c5fce624083e92546a8545e7e5

Quoted 5 lines

Yeah, it's pretty sad how few commits some of these filesystems have had in recent years. One can argue that they're stable and don't need to be fixed because they aren't broken, but I find it hard to believe that any of them were better-implemented than ext2 which still sees regular bugfixes.

Regarding affs there is a severe issue which is not in affs itself but in the handling of Rigid Disk Block (RDB) partitions, the Amiga partitioning standard, which is far more advanced than MBR: It overruns for 2 TB or larger drives and then wraps over to the beginning of the drive – I bet you can imagine what happens if you write to an area larger than 2 TB. I learned this with an external 2TB RDB partitioned harddisk back then, which I used for Sam440ep (a kind of successor for old, classic Amiga hardware) backup + some Linux related stuff in another partition.

Joanne Dow, a developer who developed hdwrench.library which HDToolBox uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then, but never officially put it officially through upstreaming as I offered to make a good description and upstream it through Jens Axboe.

I may take this as a reason to… actually follow through this time, hopefully remembering all the details in order to provide a meaningful patch description – but I think mostly I can do just careful copy and paste. Even tough I believe Joanne Dow´s fix only fixed my bug report 43511, but not 43511 which is more about a safeguarding issue in case of future overflows, I still think it would be good to go in in case affs + RDB stays in their current places.

However, in case you move affs to staging, I may be less motivated to do so, but then I suggest you also move RDB partitioning support to staging, cause this is the one that is known to be dangerously badly for 2 TB or larger disks. And yeah, I admit I did not follow through with having that patch upstreamed. Probably I did not want to be responsible in case my description would not have been absolutely accurate or the patch breaks something else. I do not have that 2 TB drive anymore and don´t feel like setting one up in a suitable way in order to go about this patch, but my testing back then was quite elaborate and I still feel pretty confident about it.

I totally get your motivation, but I would find it somewhat sad to see the filesystems you mentioned go into staging. However, as I just shown clearly, for the user it may be better, cause there may be unfixed dangerous bugs.

FUSE might be an interesting approach, but I bet it will not solve the maintenance issue. If there is no one maintaining it in the kernel, I think its unlikely to find someone adapting it to be a FUSE filesystem and maintaining it. And then I am not aware of FUSE based partitioning support. (And I think think ideally we´d had a microkernel and run all filesystems in userspace processes with a defined set of privileges, but that is simply not Linux as it is.)

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://lkml.org/lkml/2012/6/17/6

Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://bugzilla.kernel.org/show_bug.cgi?id=43511

I forward the relevant mail of Joanne, in

https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7

I even have the patch in diff format. And I just checked, the issue is still unpatched as of 4.16.3.

(adding debian-68k)

Hi Matthew!

Quoted 1 line · Martin Steigerwald

You probably put your stick into a cave with ancient sleeping dragons :)

Indeed.

Quoted 5 lines

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

The answer is that we are still very much actively using RDB and AFFS supoort in the Linux kernel and if you were to remove it, you would directly hit users.

I know it may sound crazy, but the Linux/m68k port (Atari, Mac, Amiga etc) is a very actively used and maintained port which just recently received three new drivers:

Quoted 3 lines

https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.18/scsi-queue&id=3109e5ae0311e937d49a5325134e50b742ac5f4a
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=861928f4e60e826cd8871c0c37f4b3d825b8d81d
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/pata_gayle.c?id=9ab27d1d35fda0c5fce624083e92546a8545e7e5

The community around the m68k CPU is constantly developing new hardware (new accelerator boards, networking cards, IDE controllers etc for the Amiga and so on). So, the community and the port are anything but dead.

Quoted 5 lines

Quoted 5 lines

Yeah, it's pretty sad how few commits some of these filesystems have had in recent years. One can argue that they're stable and don't need to be fixed because they aren't broken, but I find it hard to believe that any of them were better-implemented than ext2 which still sees regular bugfixes.

Exactly. It works fine as is:

There is nothing at the moment that needs fixing.

Quoted 3 lines

The answer is that we are still very much actively using RDB and AFFS supoort in the Linux kernel and if you were to remove it, you would directly hit users.

Based on that I think removing affs will not happen, but the upstream maintenance status should be updated accordingly.

Quoted 1 line

Exactly. It works fine as is:

...

Quoted 1 line

There is nothing at the moment that needs fixing.

So, I'm willing to act as upstream maintainer for affs, send pull requests with fixes if you ever need that (unless you find someone else).

Exactly. It works fine as is:

root@elgar:~> uname -a
Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52 NZDT 2018 m68k GNU/Linux root@elgar:~> mount /dev/sda1 /mnt -taffs
root@elgar:~> ls -l /mnt | head
total 0
drwx------ 1 root root 0 Mar 30 2001 Alt
-rw------- 1 root root 1352 Mar 27 1997 Alt.info

drwx------ 1 root root      0 Nov 16 14:39 C
drwx------ 1 root root      0 Mar 27  1997 CS_Fonts
drwx------ 1 root root      0 Mar 27  1997 Classes
-rwx------ 1 root root   1132 Aug 14  1996 Classes.info
drwx------ 1 root root      0 Feb 10  2004 Commodities
-rw------- 1 root root    628 Jan 14  2002 Commodities.info
drwx------ 1 root root      0 Apr 10  1999 CyberTools

root@elgar:~> mount |grep affs
/dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:) root@elgar:~>

There is nothing at the moment that needs fixing.

Quoted 1 line

There is nothing at the moment that needs fixing.

Funny, that... I'd been going through the damn thing for the last week or so; open-by-fhandle/nfs export support is completely buggered. And as for the rest... the least said about the error handling, the better - something like rename() hitting an IO error (read one) can not only screw the on-disk data into the ground, it can do seriously bad things to kernel data structures.

Quoted 2 lines · Al Viro

Quoted 1 line

There is nothing at the moment that needs fixing.

Quoted 6 lines

Funny, that... I'd been going through the damn thing for the last week or so; open-by-fhandle/nfs export support is completely buggered. And as for the rest... the least said about the error handling, the better - something like rename() hitting an IO error (read one) can not only screw the on-disk data into the ground, it can do seriously bad things to kernel data structures.

... and while we are at it, consider the following:

I'm fixing that pile of crap (along with the NFS exports
one and, hopefully, rename mess as well). HOWEVER, I am not going to take over the damn thing - David has violated the 11th commandment (Thou Shalt Never Volunteer), so he gets to joy of learning that codebase and taking care of it from now on.

I'm fixing that pile of crap (along with the NFS exports
one and, hopefully, rename mess as well). HOWEVER, I am not going to take over the damn thing - David has violated the 11th commandment (Thou Shalt Never Volunteer), so he gets to joy of learning that codebase and taking care of it from now on.

Quoted 5 lines · Al Viro

I'm fixing that pile of crap (along with the NFS exports
one and, hopefully, rename mess as well). HOWEVER, I am not going to take over the damn thing - David has violated the 11th commandment (Thou Shalt Never Volunteer), so he gets to joy of learning that codebase and taking care of it from now on.

Same scenario on link(2) ends up with
* ST_LINKFILE created, inserted into the link chain and left around, without being present in any hash chain
* target becoming positive hashed dentry, as if link(2) has succeeded, so dcache lookups will be finding it (for a while)
* unlink(2) on source will have very interesting effects, what with the attempts to move ST_FILE entry into the directory presumed to contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.

Quoted 8 lines · Al Viro

Quoted 5 lines · Al Viro

I'm fixing that pile of crap (along with the NFS exports
one and, hopefully, rename mess as well). HOWEVER, I am not going to take over the damn thing - David has violated the 11th commandment (Thou Shalt Never Volunteer), so he gets to joy of learning that codebase and taking care of it from now on.

Quoted 8 lines

Same scenario on link(2) ends up with
* ST_LINKFILE created, inserted into the link chain and left around, without being present in any hash chain
* target becoming positive hashed dentry, as if link(2) has succeeded, so dcache lookups will be finding it (for a while)
* unlink(2) on source will have very interesting effects, what with the attempts to move ST_FILE entry into the directory presumed to contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.

Oh, lovely... Looks like that thing wants the hash chains sorted by block number. affs_insert_hash() doesn't give a toss - it always adds to the very end of chain.

Maintaining that requirement (and I can understand the rationale - they don't want too many back-and-forth seeks on directory lookups) is going to be great fun on rename(), especially for the case when the target of rename happens to be a primary name for a file with additional hardlinks.

I think I see how to deal with that sanely, but... ouch.

Incidentally, we'd better verify that hash chains are not looped - as it is, there's no checks whatsoever, and it *will* happily loop if you feed it an image with such braindamage. I really hope that no fan of desktop experience has set the things up for e.g. USB sticks with that on them being recognized and automounted on insertion... --
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Al,

I don't think there is USB sticks with affs on them as yet. There isn't even USB host controller support for Amiga hardware (yet).

Last I tried USB on m68k (Atari, 060 accelerator) the desktop experience was such that I'd rather not repeat that in a hurry (and that was a simple FAT USB stick).

Michael Schmitz - 07.05.18, 04:40:

Quoted 8 lines

Al,

I don't think there is USB sticks with affs on them as yet. There isn't even USB host controller support for Amiga hardware (yet).

Last I tried USB on m68k (Atari, 060 accelerator) the desktop experience was such that I'd rather not repeat that in a hurry (and that was a simple FAT USB stick).

There is USB support available on Amiga since a long time.

On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.

On AmigaOS 4.x built-in. AmigaOS 4.x hardware like Sam boards from Acube Systems have USB controllers that work out of the bux.

And I am pretty sure, you can also tell it to use Amiga Fast Filesystem (on Linux affs) on an USB stick. Also you can plug in an external harddisk with RDB partitions and whatever filesystems you wish.

Quoted 7 lines

Quoted 6 lines

I don't think there is USB sticks with affs on them as yet. There isn't even USB host controller support for Amiga hardware (yet).

Last I tried USB on m68k (Atari, 060 accelerator) the desktop experience was such that I'd rather not repeat that in a hurry (and that was a simple FAT USB stick).

Quoted 1 line

There is USB support available on Amiga since a long time.

Good to hear that. I stand corrected.

Quoted 1 line

On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.

Haven't seen a Linux driver for that 'some USB card' yet.

Quoted 1 line

Haven't seen a Linux driver for that 'some USB card' yet.

There is for example RapidRoad (<http://wiki.icomp.de/wiki/RapidRoad>) which is an addon for the X-Surf 100 NIC (<http://wiki.icomp.de/wiki/X-Surf-100>).
Adrian already has this addon module, I will buy one this year as well.

Quoted 1 line

There is nothing at the moment that needs fixing.

Funny, that... I'd been going through the damn thing for the last week or so; open-by-fhandle/nfs export support is completely buggered. And as for the rest... the least said about the error handling, the better - something like rename() hitting an IO error (read one) can not only screw the on-disk data into the ground, it can do seriously bad things to kernel data structures.

Is there anything resembling fsck for that thing, BTW? Nevermind the repairs, just the validity checks would be nice...

https://twitter.com/mjg59/status/945779704329629697

On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz

Quoted 16 lines

Exactly. It works fine as is:

root@elgar:~> uname -a
Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52 NZDT 2018 m68k GNU/Linux root@elgar:~> mount /dev/sda1 /mnt -taffs root@elgar:~> ls -l /mnt | head
total 0
drwx------ 1 root root 0 Mar 30 2001 Alt
-rw------- 1 root root 1352 Mar 27 1997 Alt.info

drwx------ 1 root root      0 Nov 16 14:39 C
drwx------ 1 root root      0 Mar 27  1997 CS_Fonts
drwx------ 1 root root      0 Mar 27  1997 Classes
-rwx------ 1 root root   1132 Aug 14  1996 Classes.info
drwx------ 1 root root      0 Feb 10  2004 Commodities
-rw------- 1 root root    628 Jan 14  2002 Commodities.info
drwx------ 1 root root      0 Apr 10  1999 CyberTools

Quoted 5 lines

root@elgar:~> mount |grep affs
/dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:) root@elgar:~>

There is nothing at the moment that needs fixing.

Funny, that... I'd been going through the damn thing for the last week or so; open-by-fhandle/nfs export support is completely buggered. And as for the rest... the least said about the error handling, the better - something like rename() hitting an IO error (read one) can not only screw the on-disk data into the ground, it can do seriously bad things to kernel data structures.

Is there anything resembling fsck for that thing, BTW? Nevermind the repairs, just the validity checks would be nice...

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

You probably put your stick into a cave with ancient sleeping dragons :)

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

Matthew Wilcox - 26.04.18, 04:57:

Quoted 12 lines · David Sterba

Quoted 10 lines

I had similar toughts some time ago while browsing the fs/ directory.
Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

Quoted 5 lines

Yeah, it's pretty sad how few commits some of these filesystems have had in recent years. One can argue that they're stable and don't need to be fixed because they aren't broken, but I find it hard to believe that any of them were better-implemented than ext2 which still sees regular bugfixes.

Regarding affs there is a severe issue which is not in affs itself but in the handling of Rigid Disk Block (RDB) partitions, the Amiga partitioning standard, which is far more advanced than MBR: It overruns for 2 TB or larger drives and then wraps over to the beginning of the drive – I bet you can imagine what happens if you write to an area larger than 2 TB. I learned this with an external 2TB RDB partitioned harddisk back then, which I used for Sam440ep (a kind of successor for old, classic Amiga hardware) backup + some Linux related stuff in another partition.

Joanne Dow, a developer who developed hdwrench.library which HDToolBox uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then, but never officially put it officially through upstreaming as I offered to make a good description and upstream it through Jens Axboe.

I may take this as a reason to… actually follow through this time, hopefully remembering all the details in order to provide a meaningful patch description – but I think mostly I can do just careful copy and paste. Even tough I believe Joanne Dow´s fix only fixed my bug report 43511, but not 43511 which is more about a safeguarding issue in case of future overflows, I still think it would be good to go in in case affs + RDB stays in their current places.

However, in case you move affs to staging, I may be less motivated to do so, but then I suggest you also move RDB partitioning support to staging, cause this is the one that is known to be dangerously badly for 2 TB or larger disks. And yeah, I admit I did not follow through with having that patch upstreamed. Probably I did not want to be responsible in case my description would not have been absolutely accurate or the patch breaks something else. I do not have that 2 TB drive anymore and don´t feel like setting one up in a suitable way in order to go about this patch, but my testing back then was quite elaborate and I still feel pretty confident about it.

I totally get your motivation, but I would find it somewhat sad to see the filesystems you mentioned go into staging. However, as I just shown clearly, for the user it may be better, cause there may be unfixed dangerous bugs. FUSE might be an interesting approach, but I bet it will not solve the maintenance issue. If there is no one maintaining it in the kernel, I think its unlikely to find someone adapting it to be a FUSE filesystem and maintaining it. And then I am not aware of FUSE based partitioning support. (And I think think ideally we´d had a microkernel and run all filesystems in userspace processes with a defined set of privileges, but that is simply not Linux as it is.)

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://lkml.org/lkml/2012/6/17/6

Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://bugzilla.kernel.org/show_bug.cgi?id=43511

I forward the relevant mail of Joanne, in

https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7

I even have the patch in diff format. And I just checked, the issue is still unpatched as of 4.16.3.

---------- Weitergeleitete Nachricht ----------

Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
Datum: Montag, 18. Juni 2012, 03:28:48 CEST
Von: jdow <[email protected]>
An: Martin Steigerwald <[email protected]>
Kopie: Geert Uytterhoeven <[email protected]>, linux- [email protected], Jens Axboe <[email protected]>, linux- [email protected]

Quoted 17 lines · Martin Steigerwald

Am Sonntag, 17. Juni 2012 schrieb jdow:

Quoted 2 lines · Geert Uytterhoeven

Quoted 1 line

On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald

Quoted 13 lines · <[email protected]> wrote:

Quoted 13 lines

Quoted 13 lines

Am Sonntag, 17. Juni 2012 schrieb jdow:

| JXFS 64 bit file system
|
| With AmigaOS 4.x a new file system has been introduced called
| JXFS. It is a totally new 64 bit file system that supports
| partitions up to 16 TB in size. It is a modern journalling file
| system, which means that it reduces data loss if data writes to
| the disk are interrupted. It is the fastest and most reliable
| file system ever created for AmigaOS.

http://www.amigaos.net/content/1/features

Well I asked AmigaOS 4 developers about this issue as well. Lets

see

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

what they say about 2 TB limits.

16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?

block/partitions/amiga.c reads the block size from RigidDiskBlock.rdb_BlockBytes,
but after conversion to 512-byte blocks, all further calculations

are

Quoted 10 lines

Quoted 5 lines

Quoted 5 lines

done on "int", so it will overflow for disks larger than 2 TiB.

Note that in your profile-binary.img, the field is 0x200, i.e. 512 bytes per block,
so I'll have to get a deeper look into your RDB first...

[…]

Quoted 2 lines

Note that the work I did on the Linux RDB code eons ago took full cognizance of the physical and virtual block sizes.

[…]

Quoted 1 line

I've asked Martin for a digital copy of his RDBs and what he thinks

the

Quoted 8 lines

Quoted 2 lines

partition(s) should look like. I should also be told whether the disk is supposed to be solely Amiga OSs or not. I gather it's not.

Its all in the bug report. profile-binary.img should be it.

Its small so I attach it.

Meanwhile I try to get the currently supported maximum disk size out

from

Quoted 1 line

the OS 4 developers. Maybe JXFS is playing tricks that other

filesystems do

Quoted 3 lines

not play or simply has a different implementation.

Thanks,

I've sent Jens a proposed fix changing these lines in amiga.c. ===8<--- was

         struct PartitionBlock *pb;
         int start_sect, nr_sects, blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
         struct PartitionBlock *pb;
         sector_t start_sect, nr_sects;
         int blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */

===8<---

(I'm working without proper tools at the moment or I'd make a real diff. Sorry.)

This fix may not be complete. The RDBs contain provisions for describing the physical disk block size and how many physical blocks are used in a file system's logical blocks. This MAY not work quite right large physical
block sizes.

{^_^} Joanne Dow
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-------------------------------------------------------------
[…]

Quoted 7 lines

As long as there's someone who can be pestered with bugs, I don't see the need to push their baby out of the tree. I'm a bit more nervous about hfsplus; if it has users and no maintainer, those users are at risk.

Perhaps we could advertise on Craigslist or something ... maintainer wanted for LTR.

While non-native Linux filesystem support (e.g. affs/isofs/...) could be handled by FUSE

moving RDB partition support to staging is not an option, as it is the only partitioning scheme that Amigas can boot from.

If there are bugs in the RDB parser that people run into, they should be fixed. If there are limitations in the RDB format on large disks, that's still not a reason to move it to staging (hi msdos partitioning!).

Quoted 3 lines · Geert Uytterhoeven

While non-native Linux filesystem support (e.g. affs/isofs/...) could be handled by FUSE

Moving to FUSE is a great divide-and-conquer strategy for those who just want the code to die and don't care about any of the data in that format.

If there is a maintainence burden that can be shared then it should be shared -- until it can be established that there is no data of value in that format.

Quoted 3 lines

moving RDB partition support to staging is not an option, as it is the only partitioning scheme that Amigas can boot from.

Whether or not the original hardware is in use is mostly irrelevant.

As long as the old format is accessible using current hardware, the data in that format remains accessible (to archivists, to curators, to your decendents, etc).

Quoted 4 lines

If there are bugs in the RDB parser that people run into, they should be fixed. If there are limitations in the RDB format on large disks, that's still not a reason to move it to staging (hi msdos partitioning!).

Hi Martin,

CC jdow

Quoted 7 lines · Martin Steigerwald

You probably put your stick into a cave with ancient sleeping dragons :)

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

While non-native Linux filesystem support (e.g. affs/isofs/...) could be handled by FUSE, moving RDB partition support to staging is not an option, as it is the only partitioning scheme that Amigas can boot from.

If there are bugs in the RDB parser that people run into, they should be fixed.
If there are limitations in the RDB format on large disks, that's still not a reason to move it to staging (hi msdos partitioning!).

Quoted 169 lines

Matthew Wilcox - 26.04.18, 04:57:

Quoted 12 lines · David Sterba

Quoted 10 lines

I had similar toughts some time ago while browsing the fs/ directory.
Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

Quoted 5 lines

Yeah, it's pretty sad how few commits some of these filesystems have had in recent years. One can argue that they're stable and don't need to be fixed because they aren't broken, but I find it hard to believe that any of them were better-implemented than ext2 which still sees regular bugfixes.

Regarding affs there is a severe issue which is not in affs itself but in the handling of Rigid Disk Block (RDB) partitions, the Amiga partitioning standard, which is far more advanced than MBR: It overruns for 2 TB or larger drives and then wraps over to the beginning of the drive – I bet you can imagine what happens if you write to an area larger than 2 TB. I learned this with an external 2TB RDB partitioned harddisk back then, which I used for Sam440ep (a kind of successor for old, classic Amiga hardware) backup + some Linux related stuff in another partition.

Joanne Dow, a developer who developed hdwrench.library which HDToolBox uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then, but never officially put it officially through upstreaming as I offered to make a good description and upstream it through Jens Axboe.

I may take this as a reason to… actually follow through this time, hopefully remembering all the details in order to provide a meaningful patch description – but I think mostly I can do just careful copy and paste. Even tough I believe Joanne Dow´s fix only fixed my bug report 43511, but not 43511 which is more about a safeguarding issue in case of future overflows, I still think it would be good to go in in case affs + RDB stays in their current places.

However, in case you move affs to staging, I may be less motivated to do so, but then I suggest you also move RDB partitioning support to staging, cause this is the one that is known to be dangerously badly for 2 TB or larger disks. And yeah, I admit I did not follow through with having that patch upstreamed. Probably I did not want to be responsible in case my description would not have been absolutely accurate or the patch breaks something else. I do not have that 2 TB drive anymore and don´t feel like setting one up in a suitable way in order to go about this patch, but my testing back then was quite elaborate and I still feel pretty confident about it.

I totally get your motivation, but I would find it somewhat sad to see the filesystems you mentioned go into staging. However, as I just shown clearly, for the user it may be better, cause there may be unfixed dangerous bugs. FUSE might be an interesting approach, but I bet it will not solve the maintenance issue. If there is no one maintaining it in the kernel, I think its unlikely to find someone adapting it to be a FUSE filesystem and maintaining it. And then I am not aware of FUSE based partitioning support. (And I think think ideally we´d had a microkernel and run all filesystems in userspace processes with a defined set of privileges, but that is simply not Linux as it is.)

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://lkml.org/lkml/2012/6/17/6

Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://bugzilla.kernel.org/show_bug.cgi?id=43511

I forward the relevant mail of Joanne, in

https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7

I even have the patch in diff format. And I just checked, the issue is still unpatched as of 4.16.3.

---------- Weitergeleitete Nachricht ----------

Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
Datum: Montag, 18. Juni 2012, 03:28:48 CEST
Von: jdow <[email protected]>
An: Martin Steigerwald <[email protected]>
Kopie: Geert Uytterhoeven <[email protected]>, linux- [email protected], Jens Axboe <[email protected]>, linux- [email protected]

Quoted 17 lines · Martin Steigerwald

Am Sonntag, 17. Juni 2012 schrieb jdow:

Quoted 2 lines · Geert Uytterhoeven

Quoted 1 line

On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald

Quoted 13 lines · <[email protected]> wrote:

Quoted 13 lines

Quoted 13 lines

Am Sonntag, 17. Juni 2012 schrieb jdow:

| JXFS 64 bit file system
|
| With AmigaOS 4.x a new file system has been introduced called
| JXFS. It is a totally new 64 bit file system that supports
| partitions up to 16 TB in size. It is a modern journalling file
| system, which means that it reduces data loss if data writes to
| the disk are interrupted. It is the fastest and most reliable
| file system ever created for AmigaOS.

http://www.amigaos.net/content/1/features

Well I asked AmigaOS 4 developers about this issue as well. Lets

see

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

what they say about 2 TB limits.

16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?

block/partitions/amiga.c reads the block size from RigidDiskBlock.rdb_BlockBytes,
but after conversion to 512-byte blocks, all further calculations

are

Quoted 10 lines

Quoted 5 lines

Quoted 5 lines

done on "int", so it will overflow for disks larger than 2 TiB.

Note that in your profile-binary.img, the field is 0x200, i.e. 512 bytes per block,
so I'll have to get a deeper look into your RDB first...

[…]

Quoted 2 lines

Note that the work I did on the Linux RDB code eons ago took full cognizance of the physical and virtual block sizes.

[…]

Quoted 1 line

I've asked Martin for a digital copy of his RDBs and what he thinks

the

Quoted 8 lines

Quoted 2 lines

partition(s) should look like. I should also be told whether the disk is supposed to be solely Amiga OSs or not. I gather it's not.

Its all in the bug report. profile-binary.img should be it.

Its small so I attach it.

Meanwhile I try to get the currently supported maximum disk size out

from

Quoted 1 line

the OS 4 developers. Maybe JXFS is playing tricks that other

filesystems do

Quoted 3 lines

not play or simply has a different implementation.

Thanks,

I've sent Jens a proposed fix changing these lines in amiga.c. ===8<--- was

         struct PartitionBlock *pb;
         int start_sect, nr_sects, blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
         struct PartitionBlock *pb;
         sector_t start_sect, nr_sects;
         int blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */

===8<---

(I'm working without proper tools at the moment or I'd make a real diff. Sorry.)

This fix may not be complete. The RDBs contain provisions for describing the physical disk block size and how many physical blocks are used in a file system's logical blocks. This MAY not work quite right large physical
block sizes.

Quoted 16 lines

{^_^} Joanne Dow
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-------------------------------------------------------------
[…]

Quoted 7 lines

As long as there's someone who can be pestered with bugs, I don't see the need to push their baby out of the tree. I'm a bit more nervous about hfsplus; if it has users and no maintainer, those users are at risk.

Perhaps we could advertise on Craigslist or something ... maintainer wanted for LTR.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 3 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.

If memory serves the RDBs think in blocks rather than bytes so it should work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 15 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

Quoted 8 lines

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

Quoted 37 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.

If memory serves the RDBs think in blocks rather than bytes so it should work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 15 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

Quoted 8 lines

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 55 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

Quoted 31 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.

If memory serves the RDBs think in blocks rather than bytes so it should work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 20 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

Joanne.

jdow - 26.06.18, 07:17:

Quoted 9 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Quoted 4 lines

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

Heh… :)

Well, all I can say it that it just worked on AmigaOS 4. I did not see any data corruption in any of the filesystems. Well as far as I have been able to check. There has been no repair tool for JXFS I think. But as I migrated the data on it to SFS, I was able to copy everything.

Famous last words.

Well especially the disk size was detected properly and there was no overflow like on Linux. So I´d say rather have on error less than one error more.

Of course, it could also be an option to outright *refuse* to detect such disks with a big fat warning in kernel log that it may unsafe. But overflowing and thus on writes overwriting existing data is not safe.

I do think it is safe enough, but what I do know about the internals about RDB (more than the average user certainly, but not as much as you or some other AmigaOS developer who digged deeper into that).

So in case you´d rather see Linux to refuse to handle disks like that, that would also be fine with me. Just do not handle them in the broken way that they are handled in Linux now. I.e. do not deliberately corrupt things as in "Its dangerous, so let´s overwrite data straight away, so the user gets it." :)

Anyway, in my opinion RDB still just so much more advanced than MBR and in some parts even on par with the much later GPT. With some limitations it is a quite brilliant partition format, if you ask me.

Quoted 58 lines

{^_^}

Quoted 55 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

Quoted 31 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.

If memory serves the RDBs think in blocks rather than bytes so it should work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 20 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

[…]

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 66 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

   Michael

Quoted 43 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 25 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Cheers,

    Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 86 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 66 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

   Michael

Quoted 43 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 25 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

If it is not backwards compatible I for one would refuse to use it. And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 36 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Quoted 100 lines

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 93 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 17 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Quoted 55 lines

Cheers,

Michael

Quoted 47 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 22 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

Michael

Quoted 155 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 36 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Quoted 100 lines

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 93 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 17 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Quoted 55 lines

Cheers,

Michael

Quoted 47 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 22 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o} Joanne "Said enough, she has" Dow

Quoted 146 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

Michael

Quoted 108 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 93 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 49 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a
change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 27 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)

sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 62 lines

Quoted 62 lines

Quoted 51 lines

Quoted 51 lines

Quoted 50 lines

Quoted 50 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it
should
work up to 2 gigablocks whatever your block size is. 512 blocks is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 19 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow�s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 6 lines

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Dear Joanne.

jdow - 27.06.18, 08:24:

Quoted 22 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

Quoted 5 lines

However, consider my inputs as advice from an adult who has seen the Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

I am pretty confident that larger than 2 TiB disks are fully supported within AmigaOS 4, as I outlined in my other mail.

So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in 2012 already *just* fine. I even found I had the same questions back then, and researched it. Which lead to this official article back then:

http://wiki.amigaos.net/wiki/RDB

I am also pretty sure that AmigaOS still uses RDB as partitioning format. They support MBR. I don´t think AmigaOS 4.1 supports GPT. Whether to implement that of course is the decision of AmigaOS 4 development team. I am no longer a member of it since some time.

Linux m68k should already be able to use disks in GPT format, but you likely won´t be able to read them in AmigaOS, unless there is some third party support for it meanwhile.

Thanks,
Martin

Quoted 150 lines

{o.o} Joanne "Said enough, she has" Dow

Quoted 146 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

Michael

Quoted 108 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 93 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 49 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a
change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 27 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)

sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 62 lines · wrote:

Quoted 62 lines

Quoted 62 lines

Quoted 51 lines

Quoted 51 lines

Quoted 50 lines

Quoted 50 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it
should
work up to 2 gigablocks whatever your block size is. 512 blocks is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 19 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 6 lines

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The issue is what happens when one of those disks appears on a 3.1 system. {^_^}

Quoted 305 lines

Dear Joanne.

jdow - 27.06.18, 08:24:

Quoted 29 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features?
(Loading a filesystem or DriveInit code from RDBs is just asking for
a nearly impossible to remove malware infection.) Furthermore, any 32
bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably
quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB
reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it
detects within the RDBs and return an error such as to leave the disk
unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

Quoted 8 lines

However, consider my inputs as advice from an adult who has seen the
Amiga Elephant so to speak. I am not trying to assert any control. Do
as you wish; but, I would plead with you to avoid ANY chance you can
for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

I am pretty confident that larger than 2 TiB disks are fully supported within AmigaOS 4, as I outlined in my other mail.

So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in 2012 already *just* fine. I even found I had the same questions back then, and researched it. Which lead to this official article back then:

http://wiki.amigaos.net/wiki/RDB

I am also pretty sure that AmigaOS still uses RDB as partitioning format. They support MBR. I don´t think AmigaOS 4.1 supports GPT. Whether to implement that of course is the decision of AmigaOS 4 development team. I am no longer a member of it since some time.

Linux m68k should already be able to use disks in GPT format, but you
likely won´t be able to read them in AmigaOS, unless there is some third party support for it meanwhile.

Thanks,
Martin

Quoted 174 lines

{o.o} Joanne "Said enough, she has" Dow

Quoted 171 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well
taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get
away with printing a big fat warning during boot, advising the user
that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to
bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition
start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the
same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian
has advertised parted as replacement for the old tools - maybe this
would make a nice test case for parted?

Cheers,

Michael

Quoted 125 lines · jdow

If it is not backwards compatible I for one would refuse to use it.
And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 109 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing
the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition
parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger
than 2 TB (which may have ramifications with filesystems on these
partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size
stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as
on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 59 lines

As long as it preserves compatibility it should be OK, I suppose.
Personally I'd make any partitioning tool front end gently force
the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad
under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make
a
change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 33 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK

(512)
sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:

[sdb]
Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means
512
byte blocks) and can be worked around by using a different block
size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others
yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 65 lines · wrote:

Quoted 65 lines

Quoted 65 lines

Quoted 53 lines

Quoted 53 lines

Quoted 52 lines

Quoted 52 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so
it
should
work up to 2 gigablocks whatever your block size is. 512 blocks
is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 19 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 6 lines

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o} Joanne "Said enough, she has" Dow

Quoted 210 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

Michael

Quoted 170 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if
it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea.
You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for
legacy disks only.

{^_^}

Quoted 42 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.

Quoted 105 lines

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 98 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 18 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Quoted 57 lines

Cheers,

Michael

Quoted 49 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 24 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 6 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing
the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If AmigaOS 4.1 supports more recent partition formats, all the better. This is all about supporting use of legacy RDB disks on Linux (though 2 TB does stretch the definition of 'legacy' a little). My interest in this is to ensure we can continue to use RDB format disks on m68k Amiga computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to make sure we translate RDB info into Linux 512-byte block offset and size numbers correctly. The kernel won't modify the RDB at all (intentionally, that is - with the correct choice of partition sizes, Martin might well have wiped out his RDB with the current version of the parser).

The choice of refusing to mount a disk (or mounting read-only) rests with the VFS drivers alone - AFFS in that case. Not touching any of that. At partition scan time, we only have the option of making the partition available (with a warning printed), or refusing to make it available to the kernel. Once it's made available, all bets are off.

From what Martin writes, his test case RDB was valid and worked as expected on
32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to handle the large offsets resulting from 2 TB disks. Not sure what safeguards are in place when connecting such a disk to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see in the partition parser - there is no other overflow I've identified. I just stated that in order to place a partition towards the end of a 2 TB disk, the offset calculation will overflow regardless of what combination of rdb->rdb_BlockBytes and sector addresses stored in the RDB (in units of 512 byte blocks) we use:

        blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;

                nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 - be32_to_cpu(pb->pb_Environment[9])) * be32_to_cpu(pb->pb_Environment[3]) * be32_to_cpu(pb->pb_Environment[5]) * blksize;
                if (!nr_sects)
                        continue;
                start_sect = be32_to_cpu(pb->pb_Environment[9]) * be32_to_cpu(pb->pb_Environment[3]) * be32_to_cpu(pb->pb_Environment[5]) * blksize;

But in the interest of avoiding any accidental use of a RDB partition where calculations currently overflow, I'll make the default behaviour to bail out (instead of using wrong offset or size as we currently do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though I still can't see what harm should result from now translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1 TB is a matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent accidental use of AFFS filesystems on RDB disks which I suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the partitioning tools bypass the partition structure discovered by the kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions, for a disk 2 TB or larger that was partitioned and used on a recent version or AmigaOS supporting such large disks, I'd consider omitting the 'eat_my_RDB_disk' boot option, and just bail out as the only safe option.

Cheers,

    Michael

Am 27.06.2018 um 18:24 schrieb jdow:

Quoted 240 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o}   Joanne "Said enough, she has" Dow

Quoted 210 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

   Michael

Quoted 170 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if
it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea.
You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for
legacy disks only.

{^_^}

Quoted 147 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.

Cheers,

      Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 98 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 75 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

     Michael

Quoted 49 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 30 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 6 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing
the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael, as long as m68k only supports int fseek( int ) 4 GB * block size is your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with larger disks. RDBs could include RDSK and a new set of other blocks that replace the last two characters with "64" and use __int64 where needed in the various values. That way a clever disk partitioner could give allow normal (32 bit) RDB definitions where possible. Then at least SOME of the disk could be supported AND a very clever filesystem that abstracts very large disks properly could give access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.

I could be wrong but I do not think RDBs could be safely modified any other way to work. And, trust me as I bet this is still true, you will need a SERIOUSLY good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that crashes on anything newer. Then Joe got far enough something writes to the disk and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to facts in such cases. Hell, some of them never listened to facts about an incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.

{o.o}

Quoted 342 lines · Michael Schmitz

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If AmigaOS 4.1 supports more recent partition formats, all the better. This is all about supporting use of legacy RDB disks on Linux (though 2 TB does stretch the definition of 'legacy' a little). My interest in this is to ensure we can continue to use RDB format disks on m68k Amiga computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to make sure we translate RDB info into Linux 512-byte block offset and size numbers correctly. The kernel won't modify the RDB at all (intentionally, that is - with the correct choice of partition sizes, Martin might well have wiped out his RDB with the current version of the parser).

The choice of refusing to mount a disk (or mounting read-only) rests with the VFS drivers alone - AFFS in that case. Not touching any of that. At partition scan time, we only have the option of making the partition available (with a warning printed), or refusing to make it available to the kernel. Once it's made available, all bets are off.

From what Martin writes, his test case RDB was valid and worked as
expected on 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to handle the large offsets resulting from 2 TB disks. Not sure what safeguards are in place when connecting such a disk to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see in the partition parser - there is no other overflow I've identified. I just stated that in order to place a partition towards the end of a 2 TB disk, the offset calculation will overflow regardless of what combination of rdb->rdb_BlockBytes and sector addresses stored in the RDB (in units of 512 byte blocks) we use:

         blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;


                 nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
                             be32_to_cpu(pb->pb_Environment[9])) *
                            be32_to_cpu(pb->pb_Environment[3]) *
                            be32_to_cpu(pb->pb_Environment[5]) *
                            blksize;
                 if (!nr_sects)
                         continue;
                 start_sect = be32_to_cpu(pb->pb_Environment[9]) *
                              be32_to_cpu(pb->pb_Environment[3]) *
                              be32_to_cpu(pb->pb_Environment[5]) *
                              blksize;

But in the interest of avoiding any accidental use of a RDB partition where calculations currently overflow, I'll make the default behaviour to bail out (instead of using wrong offset or size as we currently do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though I still can't see what harm should result from now translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1 TB is a matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent accidental use of AFFS filesystems on RDB disks which I suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the partitioning tools bypass the partition structure discovered by the kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions, for a disk 2 TB or larger that was partitioned and used on a recent version or AmigaOS supporting such large disks, I'd consider omitting the 'eat_my_RDB_disk' boot option, and just bail out as the only safe option.

Cheers,

Michael

Am 27.06.2018 um 18:24 schrieb jdow:

Quoted 261 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o} Joanne "Said enough, she has" Dow

Quoted 231 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

Michael

Quoted 191 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if
it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea.
You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
only system. If it is for your personal use then you're entirely free to
reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for
legacy disks only.

{^_^}

Quoted 52 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that may
have to deal with large disks) and fixes Martin's bug, so appears to be
the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is
preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't
say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a
RDB image from a disk known to currently work under Linux (thanks Geert
for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.

Quoted 114 lines

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 107 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad under
1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining
I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 22 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)

sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Quoted 59 lines

Cheers,

Michael

Quoted 51 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 25 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 7 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing
the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Joanne,

Linux on m68k has supported lseek64 (or llseek) for a long time (from glibc version 2.1 according to what I found). About the only area where we are limited by 32 bits is the virtual memory size.

I'm not proposing to modify the RDB format definition, though an extension to store 64 bit offsets separate from the 32 bit ones would be one way to make certain such disks are safe to use on 3.1 and earlier versions of AmigaOS. (Another one would be to modify the disk drivers on older versions to do the offset calculation in 64 bit, and check for overflow just as we do here. Not sure whether that's feasible. And as you so eloquently describe, we can't rely on users listening.)

Either way, we need the cooperation of partitioning tool writers to ensure partition information that is prone to overflows is never stored in the 32 bit, classic RDB. That appears to have failed already, as Martin's experience illustrates.

I'm only concerned with fixing the (dangerous) but in the Linux partition format parser for RDB. Refusing to use any partitions that will cause havoc on old AmigaOS versions is all I can do to try and get the users' attention.

Your warning makes me wonder whether the log message should just say 'report this bug to [email protected]' to at least try and educate any that respond about the dangers of their partitioning scheme before telling them about the override option. Problem with that is, in about three years no one will remember any of this ...

Cheers,

    Michael

Am 28.06.2018 um 15:44 schrieb jdow:

Quoted 369 lines

Michael, as long as m68k only supports int fseek( int ) 4 GB * block size is your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with larger disks. RDBs could include RDSK and a new set of other blocks that replace the last two characters with "64" and use __int64 where needed in the various values. That way a clever disk partitioner could give allow normal (32 bit) RDB definitions where possible. Then at least SOME of the disk could be supported AND a very clever filesystem that abstracts very large disks properly could give access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.

I could be wrong but I do not think RDBs could be safely modified any other way to work. And, trust me as I bet this is still true, you will need a SERIOUSLY good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that crashes on anything newer. Then Joe got far enough something writes to the disk and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to facts in such cases. Hell, some of them never listened to facts about an incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.

{o.o}

Quoted 342 lines · Michael Schmitz

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If AmigaOS 4.1 supports more recent partition formats, all the better. This is all about supporting use of legacy RDB disks on Linux (though 2 TB does stretch the definition of 'legacy' a little). My interest in this is to ensure we can continue to use RDB format disks on m68k Amiga computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to make sure we translate RDB info into Linux 512-byte block offset and size numbers correctly. The kernel won't modify the RDB at all (intentionally, that is - with the correct choice of partition sizes, Martin might well have wiped out his RDB with the current version of the parser).

The choice of refusing to mount a disk (or mounting read-only) rests with the VFS drivers alone - AFFS in that case. Not touching any of that. At partition scan time, we only have the option of making the partition available (with a warning printed), or refusing to make it available to the kernel. Once it's made available, all bets are off.

 From what Martin writes, his test case RDB was valid and worked as expected on 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to handle the large offsets resulting from 2 TB disks. Not sure what safeguards are in place when connecting such a disk to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see in the partition parser - there is no other overflow I've identified. I just stated that in order to place a partition towards the end of a 2 TB disk, the offset calculation will overflow regardless of what combination of rdb->rdb_BlockBytes and sector addresses stored in the RDB (in units of 512 byte blocks) we use:

         blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;

                 nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 - be32_to_cpu(pb->pb_Environment[9])) * be32_to_cpu(pb->pb_Environment[3]) * be32_to_cpu(pb->pb_Environment[5]) * blksize;
                 if (!nr_sects)
                         continue;
                 start_sect = be32_to_cpu(pb->pb_Environment[9]) * be32_to_cpu(pb->pb_Environment[3]) * be32_to_cpu(pb->pb_Environment[5]) * blksize;

But in the interest of avoiding any accidental use of a RDB partition where calculations currently overflow, I'll make the default behaviour to bail out (instead of using wrong offset or size as we currently do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though I still can't see what harm should result from now translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1 TB is a matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent accidental use of AFFS filesystems on RDB disks which I suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the partitioning tools bypass the partition structure discovered by the kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions, for a disk 2 TB or larger that was partitioned and used on a recent version or AmigaOS supporting such large disks, I'd consider omitting the 'eat_my_RDB_disk' boot option, and just bail out as the only safe option.

Cheers,

     Michael

Am 27.06.2018 um 18:24 schrieb jdow:

Quoted 261 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o}   Joanne "Said enough, she has" Dow

Quoted 231 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Cheers,

   Michael

Quoted 191 lines · jdow

If it is not backwards compatible I for one would refuse to use it. And if
it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea.
You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
only system. If it is for your personal use then you're entirely free to
reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for
legacy disks only.

{^_^}

Quoted 166 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block addresses that overflow on disks 2 TB and larger is a bug, causing the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that may
have to deal with large disks) and fixes Martin's bug, so appears to be
the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is
preserved.

Forcing larger block sizes might be a good strategy to avoid overflow issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't
say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a
RDB image from a disk known to currently work under Linux (thanks Geert
for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.

Cheers,

      Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 107 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the block size towards 8k as the disk size gets larger. The file systems may also run into 2TB issues that are not obvious. An unused blocks list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad under
1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I remember thinking about this for awhile and then determining
I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 81 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)

sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units. I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

     Michael

Quoted 51 lines · jdow

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it should
work up to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 31 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 7 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing
the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-m68k" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Anything done to RDBs for Linux must remain 100.000% compatible with existing Amiga equipment. Otherwise, what's the point of bothering to use RDBs?

That brings to the fore an interesting question. Why bother with RDBs over 2TB unless you want a disk with one single partition? This Win10 monster I am using has a modest BIOS driver partition for the OS and a giant data partition. That smaller partition would easily work with any RDB/Filesystem combination since 2.0. So there are some good workarounds that are probably "safer" and at least as flexible as RDBs, one Linux has used for a very long time, too.

As I have said, for the RDB parser fix the famndool thing. Do fix it right in such a manner that if somebody compiles it against a version with no 64 bit device code it will throw a proper overflow error and protect the user. Users are dumb. We like to think of ourselves as

smart. Let's try to be smart about this where we can so fingers can't point back at us rather than the fool that made some other error.

And do remember, I am merely (and vociferously) advising rather than dictating. You don't need my approval to proceed. I may want my name noted as an early contributor only. Meanwhile I spit out ideas as they come to me. One or more of them might be good. And offering alternatives is better than simply saying "No" most of the time.

If people are using RDBs for TB level disks I doubt they can remember which is the left shoe when they are getting dressed in the morning before going out in the yard to beat some dead horses. Or else maybe

they just want to see how far they can flog the m68k architecture as a mental challenge. In that case, taking it too seriously could hurt.

Note that I am mostly ignoring m68k Linux. People using that are hard core. People using x86/x64 Linux aren't such hard core folks. And I bet most of them want to read the disks so they can copy stuff to Amiga Forever or WinUAE running on other architectures. So TB is not likely to be an issue for them, either.

{^_^}

Quoted 256 lines · Michael Schmitz

Joanne,

Linux on m68k has supported lseek64 (or llseek) for a long time (from glibc version 2.1 according to what I found). About the only area where we are limited by 32 bits is the virtual memory size.

I'm not proposing to modify the RDB format definition, though an extension to store 64 bit offsets separate from the 32 bit ones would be one way to make certain such disks are safe to use on 3.1 and earlier versions of AmigaOS. (Another one would be to modify the disk drivers on older versions to do the offset calculation in 64 bit, and check for overflow just as we do here. Not sure whether that's feasible. And as you so eloquently describe, we can't rely on users listening.)

Either way, we need the cooperation of partitioning tool writers to ensure partition information that is prone to overflows is never stored in the 32 bit, classic RDB. That appears to have failed already, as Martin's experience illustrates.

I'm only concerned with fixing the (dangerous) but in the Linux partition format parser for RDB. Refusing to use any partitions that will cause havoc on old AmigaOS versions is all I can do to try and get the users' attention.

Your warning makes me wonder whether the log message should just say 'report this bug to [email protected]' to at least try and educate any that respond about the dangers of their partitioning scheme before telling them about the override option. Problem with that is, in about three years no one will remember any of this ...

Cheers,

Michael

Am 28.06.2018 um 15:44 schrieb jdow:

Quoted 220 lines

Michael, as long as m68k only supports int fseek( int ) 4 GB * block
size is your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with larger disks. RDBs could include RDSK and a new set of other blocks that replace the last two characters with "64" and use __int64 where needed in the various values. That way a clever disk partitioner could give allow normal (32 bit) RDB definitions where possible. Then at least SOME of the disk could be supported AND a very clever filesystem that abstracts very large disks properly could give access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.

I could be wrong but I do not think RDBs could be safely modified any
other way to work. And, trust me as I bet this is still true, you will need a SERIOUSLY good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that crashes on anything newer. Then Joe got far enough something writes to the disk and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to facts in such cases. Hell, some of them never listened to facts about an incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.

{o.o}

Quoted 188 lines · Michael Schmitz

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If
AmigaOS 4.1 supports more recent partition formats, all the better.
This is all about supporting use of legacy RDB disks on Linux (though
2 TB does stretch the definition of 'legacy' a little). My interest in this is to ensure we can continue to use RDB format disks on m68k Amiga computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to
make sure we translate RDB info into Linux 512-byte block offset and
size numbers correctly. The kernel won't modify the RDB at all (intentionally, that is - with the correct choice of partition sizes,
Martin might well have wiped out his RDB with the current version of
the parser).

The choice of refusing to mount a disk (or mounting read-only) rests
with the VFS drivers alone - AFFS in that case. Not touching any of
that. At partition scan time, we only have the option of making the
partition available (with a warning printed), or refusing to make it
available to the kernel. Once it's made available, all bets are off.

From what Martin writes, his test case RDB was valid and worked
as
expected on 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to handle the large offsets resulting from 2 TB
disks. Not sure what safeguards are in place when connecting such a
disk to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see in the partition parser - there is no other overflow I've identified. I just stated that in order to place a partition towards the end of a 2 TB disk, the offset calculation will overflow regardless of what combination of rdb->rdb_BlockBytes and sector addresses stored in the RDB (in units of 512 byte blocks) we use:

         blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;


                 nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
1 - be32_to_cpu(pb->pb_Environment[9])) *
be32_to_cpu(pb->pb_Environment[3]) *
be32_to_cpu(pb->pb_Environment[5]) * blksize;
                 if (!nr_sects)
                         continue;
                 start_sect = be32_to_cpu(pb->pb_Environment[9]) *
                              be32_to_cpu(pb->pb_Environment[3]) *
                              be32_to_cpu(pb->pb_Environment[5]) *
                              blksize;

But in the interest of avoiding any accidental use of a RDB partition
where calculations currently overflow, I'll make the default behaviour to bail out (instead of using wrong offset or size as we currently do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though I still can't see what harm should result from now translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1
TB is a matter for VFS experts. Bailing out on nr_sects overflowing
ought to prevent accidental use of AFFS filesystems on RDB disks which I suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the
partitioning tools bypass the partition structure discovered by the
kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions,
for a disk 2 TB or larger that was partitioned and used on a recent
version or AmigaOS supporting such large disks, I'd consider omitting
the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
option.

Cheers,

Michael

Am 27.06.2018 um 18:24 schrieb jdow:

Quoted 83 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the
Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o} Joanne "Said enough, she has" Dow

Quoted 49 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well
taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get
away with printing a big fat warning during boot, advising the user
that the oversize RDB partition(s) scanned are not compatible with
legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to
bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition
start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the
same overflow. And amiga-fdisk is indeed utterly broken and needs
updating (along with probably most legacy m68k partitioners). Adrian
has advertised parted as replacement for the old tools - maybe this
would make a nice test case for parted?

Cheers,

Michael

On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]>

Quoted 164 lines

Quoted 164 lines

Quoted 164 lines

Quoted 164 lines

Quoted 164 lines

Quoted 164 lines

If it is not backwards compatible I for one would refuse to use it.
And if
it still mattered that much to me I'd also generate a reasonable
alternative. Modifying RDBs nay not be even an approximation of a
good idea.
You'd discover that as soon as an RDB uint64_t disk is tasted by a
uint32_t
only system. If it is for your personal use then you're entirely
free to
reject my advice and are probably smart enough to keep it working for
yourself.

GPT is probably the right way to go. Preserve the ability to read
RDBs for
legacy disks only.

{^_^}

Quoted 138 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing
the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that
may
have to deal with large disks) and fixes Martin's bug, so appears
to be
the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other
than
enabling use of disk and partitions larger than 2 TB (which may have
ramifications with filesystems on these partitions). So comptibility is
preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size
stored
in the RDB would enforce use of the same block size in filesystems.
We'll have to rely on the filesystem tools to get that right, too.
Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should
allow partitions larger than 2 TB to work already (but I suspect Al
Viro
may have found a few issues when he looked at the AFFS code so I
won't
say more). Anyway partitioning tools and filesystems are unrelated to
the Linux partition parser code which is all we aim to fix in this
patch.

If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a
warning about these partitions.

I'll get this patch tested on Martin's test case image as well as
on a
RDB image from a disk known to currently work under Linux (thanks
Geert
for the losetup hint). Can't do much more without procuring a working
Amiga disk image to use with an emulator, sorry. The Amiga I plan to
use
for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 62 lines

As long as it preserves compatibility it should be OK, I suppose.
Personally I'd make any partitioning tool front end gently force the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a
block
list (OFS for sure, don't remember for the newer AFS) uses a tad
under
1% of the disk all by itself. A block bitmap is not quite so bad.
{^_-}

Just be sure you are aware of all the ramifications when you make a
change. I remember thinking about this for awhile and then determining
I REALLY did not want to think about it as my brain was getting tied
into a gordian knot.

{^_^}

Quoted 33 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK

(512)
sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:

[sdb]
Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means
512
byte blocks) and can be worked around by using a different block
size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others
yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

Quoted 40 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it
should
work up to 2 gigablocks whatever your block size is. 512 blocks is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 22 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 6 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given
there,
so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself
preparing
the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Dear Joanne.

jdow - 28.06.18, 08:39:

Quoted 3 lines

Anything done to RDBs for Linux must remain 100.000% compatible with existing Amiga equipment. Otherwise, what's the point of bothering to use RDBs?

Done to, in the sense of written to: Yes. I completely agree. But that is for amiga-fdisk and parted. And for partitioning tools on native OS.

[…]

Quoted 7 lines

That brings to the fore an interesting question. Why bother with RDBs over 2TB unless you want a disk with one single partition? This Win10 monster I am using has a modest BIOS driver partition for the OS and a giant data partition. That smaller partition would easily work with any RDB/Filesystem combination since 2.0. So there are some good workarounds that are probably "safer" and at least as flexible as RDBs, one Linux has used for a very long time, too.

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

Quoted 12 lines

Quoted 11 lines

That brings to the fore an interesting question. Why bother with RDBs
over 2TB unless you want a disk with one single partition? This Win10
monster I am using has a modest BIOS driver partition for the OS and
a giant data partition. That smaller partition would easily work with
any RDB/Filesystem combination since 2.0. So there are some good workarounds that are probably "safer" and at least as flexible as RDBs, one Linux has used for a very long time, too.

Quoted 5 lines

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

EEEEEEK! The hair on my neck is standing up straight! Have you heard of SAMBA? The linux mail server firewall etc machine has an extra 4TB disk on it as a backup for the other systems, although a piddly 4TB is small when I save the entire 3G RAID system I have. It's a proof of concept so.... A full backup on a 1gig Ethernet still takes a looooong time. But backing up even an 18GB disk on an Amiga via 100Base-t isn't too bad. And disk speeds of the era being what they were it's about all you can do anyway.

The raw, theoretical limit on the maximum device capacity is about 2^105 bytes:

32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512 bytes/sector for the HD size in struct RigidDiskBlock

jdow - 28.06.18, 12:00:

Quoted 1 line · Martin Steigerwald

[…]

Quoted 18 lines

Quoted 12 lines

Quoted 11 lines

That brings to the fore an interesting question. Why bother with RDBs
over 2TB unless you want a disk with one single partition? This Win10
monster I am using has a modest BIOS driver partition for the OS and
a giant data partition. That smaller partition would easily work with
any RDB/Filesystem combination since 2.0. So there are some good workarounds that are probably "safer" and at least as flexible as RDBs, one Linux has used for a very long time, too.

Quoted 5 lines

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

Quoted 9 lines

EEEEEEK! The hair on my neck is standing up straight! Have you heard of SAMBA? The linux mail server firewall etc machine has an extra 4TB
disk on it as a backup for the other systems, although a piddly 4TB is small when I save the entire 3G RAID system I have. It's a proof of concept so.... A full backup on a 1gig Ethernet still takes a looooong time. But backing up even an 18GB disk on an Amiga via 100Base-t isn't too bad. And disk speeds of the era being what they were it's about all you can do anyway.

Heh, the thing worked just fine in Amiga OS 4. I got away with it without an issue, until I plugged the disk to my Linux laptop and wrote data onto the Linux file system. Mind you, I think in that partition marked LNX\0 I even created a Linux LVM with pvcreate. Do you call that insane? Well it probably is. :)

And as an Amiga user I could just return to you: I clicked it, it did not warn, so all is good :)

But yeah, as mentioned I researched the topic before. And I think there
has not even been an overflow within the RDB:

Quoted 5 lines

The raw, theoretical limit on the maximum device capacity is about 2^105 bytes:

32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512 bytes/sector for the HD size in struct RigidDiskBlock

http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)

Confirmed by:

The .ADF (Amiga Disk File) format FAQ:
http://lclevy.free.fr/adflib/adf_info.html#p6

But what do I write, you know the RDB format :)

So just do the calculation in 96 Bit and you all are set :)

Now that is a reason for 128 Bit CPUs :).

Muuhahaha.

Martin Steigerwald - 28.06.18, 13:30:

Quoted 52 lines

jdow - 28.06.18, 12:00:

Quoted 1 line · Martin Steigerwald

[…]

Quoted 18 lines

Quoted 12 lines

Quoted 11 lines

That brings to the fore an interesting question. Why bother with RDBs
over 2TB unless you want a disk with one single partition? This Win10
monster I am using has a modest BIOS driver partition for the OS and
a giant data partition. That smaller partition would easily work with
any RDB/Filesystem combination since 2.0. So there are some good workarounds that are probably "safer" and at least as flexible as RDBs, one Linux has used for a very long time, too.

Quoted 5 lines

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

Quoted 9 lines

EEEEEEK! The hair on my neck is standing up straight! Have you heard of SAMBA? The linux mail server firewall etc machine has an extra 4TB
disk on it as a backup for the other systems, although a piddly 4TB is small when I save the entire 3G RAID system I have. It's a proof of concept so.... A full backup on a 1gig Ethernet still takes a looooong time. But backing up even an 18GB disk on an Amiga via 100Base-t isn't too bad. And disk speeds of the era being what they were it's about all you can do anyway.

Heh, the thing worked just fine in Amiga OS 4. I got away with it without an issue, until I plugged the disk to my Linux laptop and wrote data onto the Linux file system. Mind you, I think in that partition marked LNX\0 I even created a Linux LVM with pvcreate. Do you call that insane? Well it probably is. :)

And as an Amiga user I could just return to you: I clicked it, it did not warn, so all is good :)

But yeah, as mentioned I researched the topic before. And I think there
has not even been an overflow within the RDB:

Quoted 5 lines

The raw, theoretical limit on the maximum device capacity is about 2^105 bytes:

32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512 bytes/sector for the HD size in struct RigidDiskBlock

http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)

Quoted 8 lines

Confirmed by:

The .ADF (Amiga Disk File) format FAQ:
http://lclevy.free.fr/adflib/adf_info.html#p6

But what do I write, you know the RDB format :)

So just do the calculation in 96 Bit and you all are set :)

For sectors.

Or

3*32+9 (for 512 bytes per sector) = 105 bits

for bytes.

Quoted 3 lines

Now that is a reason for 128 Bit CPUs :).

Muuhahaha.

Joanne,

Linux on m68k has supported lseek64 (or llseek) for a long time (from glibc version 2.1 according to what I found). About the only area where we are limited by 32 bits is the virtual memory size.

I'm not proposing to modify the RDB format definition, though an extension to store 64 bit offsets separate from the 32 bit ones would be one way to make certain such disks are safe to use on 3.1 and earlier versions of AmigaOS. (Another one would be to modify the disk drivers on older versions to do the offset calculation in 64 bit, and check for overflow just as we do here. Not sure whether that's feasible. And as you so eloquently describe, we can't rely on users listening.)

Either way, we need the cooperation of partitioning tool writers to ensure partition information that is prone to overflows is never stored in the 32 bit, classic RDB. That appears to have failed already, as Martin's experience illustrates.

The raw, theoretical limit on the maximum device capacity is about 2^105 bytes:

32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512 bytes/sector for the HD size in struct RigidDiskBlock

I'm only concerned with fixing the (dangerous) but in the Linux partition format parser for RDB. Refusing to use any partitions that will cause havoc on old AmigaOS versions is all I can do to try and get the users' attention.

Your warning makes me wonder whether the log message should just say 'report this bug to [email protected]' to at least try and educate any that respond about the dangers of their partitioning scheme before telling them about the override option. Problem with that is, in about three years no one will remember any of this ...

Cheers,

Michael

Am 28.06.2018 um 15:44 schrieb jdow:

Quoted 373 lines

Michael, as long as m68k only supports int fseek( int ) 4 GB * block size is your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with larger disks. RDBs could include RDSK and a new set of other blocks that replace the last two characters with "64" and use __int64 where needed in the various values. That way a clever disk partitioner could give allow normal (32 bit) RDB definitions where possible. Then at least SOME of the disk could be supported AND a very clever filesystem that abstracts very large disks properly could give access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.

I could be wrong but I do not think RDBs could be safely modified any
other way to work. And, trust me as I bet this is still true, you will need a SERIOUSLY good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that crashes on anything newer. Then Joe got far enough something writes to the disk and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to facts in such cases. Hell, some of them never listened to facts about an incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.

{o.o}

Quoted 344 lines · Michael Schmitz

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If
AmigaOS 4.1 supports more recent partition formats, all the better. This is all about supporting use of legacy RDB disks on Linux (though
2 TB does stretch the definition of 'legacy' a little). My interest in this is to ensure we can continue to use RDB format disks on m68k Amiga computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to
make sure we translate RDB info into Linux 512-byte block offset and
size numbers correctly. The kernel won't modify the RDB at all (intentionally, that is - with the correct choice of partition sizes,
Martin might well have wiped out his RDB with the current version of
the parser).

The choice of refusing to mount a disk (or mounting read-only) rests
with the VFS drivers alone - AFFS in that case. Not touching any of that. At partition scan time, we only have the option of making the partition available (with a warning printed), or refusing to make it
available to the kernel. Once it's made available, all bets are off.

From what Martin writes, his test case RDB was valid and worked as

expected on 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to handle the large offsets resulting from 2 TB
disks. Not sure what safeguards are in place when connecting such a disk to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see in the partition parser - there is no other overflow I've identified. I just stated that in order to place a partition towards the end of a 2 TB disk, the offset calculation will overflow regardless of what combination of rdb->rdb_BlockBytes and sector addresses stored in the>>
RDB (in units of 512 byte blocks) we use:

         blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
         
                 nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
                 1 -
                 
                             be32_to_cpu(pb->pb_Environment[9])) *
                            
                            be32_to_cpu(pb->pb_Environment[3]) *
                            be32_to_cpu(pb->pb_Environment[5]) *
                            blksize;
                 
                 if (!nr_sects)
                 
                         continue;
                 
                 start_sect = be32_to_cpu(pb->pb_Environment[9]) *
                 
                              be32_to_cpu(pb->pb_Environment[3]) *
                              be32_to_cpu(pb->pb_Environment[5]) *
                              blksize;

But in the interest of avoiding any accidental use of a RDB partition
where calculations currently overflow, I'll make the default behaviour to bail out (instead of using wrong offset or size as we currently do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though I still can't see what harm should result from now translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1
TB is a matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent accidental use of AFFS filesystems on RDB disks which I suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the partitioning tools bypass the partition structure discovered by the kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions,
for a disk 2 TB or larger that was partitioned and used on a recent version or AmigaOS supporting such large disks, I'd consider omitting
the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
option.

Cheers,

Michael

Am 27.06.2018 um 18:24 schrieb jdow:

Quoted 241 lines

You allergic to using a GPT solution? It will get away from some of the evils that RDB has inherent in it because they are also features? (Loading a filesystem or DriveInit code from RDBs is just asking for a nearly impossible to remove malware infection.) Furthermore, any 32 bit system that sees an RDSK block is going to try to translate it. If you add a new RDB format you are going to get bizarre and probably quite destructive results from the mistake. Fail safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system seems utterly outlandish to me. You cited another overflow potential. There are at least three we've identified, I believe. Are you 100% sure there are no more? The specific one you mention of translating RDB to Linux has a proper solution in the RDB reader. It should recover such overflow errors in the RDB as it can with due care and polish. It should flag any other overflow error it detects within the RDBs and return an error such as to leave the disk unmounted or mounted read-only if you feel like messing up a poor sod's backups. The simple solution is to read each of the variables with the nominal RDB size and convert it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the
Amiga Elephant so to speak. I am not trying to assert any control. Do as you wish; but, I would plead with you to avoid ANY chance you can for the user to make a bonehead stupid move and lose all his treasured disk archives. Doing otherwise is very poor form.

{o.o} Joanne "Said enough, she has" Dow

Quoted 209 lines · Michael Schmitz

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well
taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get
away with printing a big fat warning during boot, advising the user
that the oversize RDB partition(s) scanned are not compatible with
legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to
bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition
start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the
same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian
has advertised parted as replacement for the old tools - maybe this
would make a nice test case for parted?

Cheers,

Michael

Quoted 162 lines · jdow

If it is not backwards compatible I for one would refuse to use it.
And if
it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a
good idea.
You'd discover that as soon as an RDB uint64_t disk is tasted by a
uint32_t
only system. If it is for your personal use then you're entirely free to
reject my advice and are probably smart enough to keep it working for
yourself.

GPT is probably the right way to go. Preserve the ability to read
RDBs for
legacy disks only.

{^_^}

Quoted 138 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing
the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that
may
have to deal with large disks) and fixes Martin's bug, so appears
to be
the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other
than
enabling use of disk and partitions larger than 2 TB (which may have
ramifications with filesystems on these partitions). So comptibility is
preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size
stored
in the RDB would enforce use of the same block size in filesystems.
We'll have to rely on the filesystem tools to get that right, too.
Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should
allow partitions larger than 2 TB to work already (but I suspect Al
Viro
may have found a few issues when he looked at the AFFS code so I
won't
say more). Anyway partitioning tools and filesystems are unrelated to
the Linux partition parser code which is all we aim to fix in this
patch.

If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a
warning about these partitions.

I'll get this patch tested on Martin's test case image as well as
on a
RDB image from a disk known to currently work under Linux (thanks
Geert
for the losetup hint). Can't do much more without procuring a working
Amiga disk image to use with an emulator, sorry. The Amiga I plan to
use
for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 62 lines

As long as it preserves compatibility it should be OK, I suppose.
Personally I'd make any partitioning tool front end gently force the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a
block
list (OFS for sure, don't remember for the newer AFS) uses a tad
under
1% of the disk all by itself. A block bitmap is not quite so bad.
{^_-}

Just be sure you are aware of all the ramifications when you make a
change. I remember thinking about this for awhile and then determining
I REALLY did not want to think about it as my brain was getting tied
into a gordian knot.

{^_^}

Quoted 33 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK

(512)
sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:

[sdb]
Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means
512
byte blocks) and can be worked around by using a different block
size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others
yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 61 lines

Quoted 61 lines

Quoted 61 lines

Quoted 61 lines

Quoted 61 lines

Quoted 60 lines

Quoted 60 lines

Quoted 45 lines

Quoted 45 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it
should
work up to 2 gigablocks whatever your block size is. 512 blocks is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 27 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 6 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there,
so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself
preparing
the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

Quoted 107 lines

If it is not backwards compatible I for one would refuse to use it. And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 93 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 49 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a
change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 27 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)

sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 59 lines

Quoted 50 lines

Quoted 50 lines

Quoted 49 lines

Quoted 49 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it
should
work up to 2 gigablocks whatever your block size is. 512 blocks is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 19 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Michael Schmitz - 27.06.18, 03:07:

Quoted 10 lines

Joanne,

As far as I have been able to test, the change is backwards compatible (RDB partitions from an old disk 80 GB disk are still recognized OK). That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have failed the current Linux RDB parser on legacy 32 bit systems is well taken though. Maybe Martin can clarify that for me - was the 2 TB disk in question ever used on a 32 bit Amiga system?

Sure! Are there actually *any* 64 bit Amiga systems? I am not completely sure architecture-wise, but the operating system is still just 32-Bit.

However, on AmigaOS officially since at least 3.5/3.9 there are various mechanisms to handle 64-bit block numbers for disk devices, called NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is part of the operating system. First via SetPatch based update to Kickstart ROM, but with AmigaOS 4.x its just included.

[1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits

(I do not agree with maximum filesystem size limits stated there for Amiga Fast Filesystem (FFS), but well, for early FFS versions they could have been true, I am pretty sure that at least with higher block sizes you can have FFS > 2 GiB in size. And I remember having had JFXS and I think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)

Quoted 7 lines

RDB disk format is meant for legacy use only, so I think we can get away with printing a big fat warning during boot, advising the user that the oversize RDB partition(s) scanned are not compatible with legacy 32 bit AmigaOS. With the proposed fix they will work under both AmigaOS 4.1 and Linux instead of truncating the first overflowing partition at disk end and trashing valid partitions that overlap, which is what Martin was after.

They are compatible. For AmigaOS 4.x I am completely sure, cause I installed and used the disk there. That was the original motivation for my Linux bug report back then: I used the disk in AmigaOS 4 *just fine* and it broke in Linux. I used Media Toolbox in order to create the RDB I posted back then.

Its difficult to find proof for my claims online, but here is an official one:

http://wiki.amigaos.net/wiki/RDB

Limits of filesystem sizes may of course be different, but here we are talking about RDB.

Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.

Hmmm, I even created the page back then. Interestingly at about the same time I made the bug report about the Linux issue. The calculations in there are not really from me. I bet I copied them over from what other AmigaOS developers wrote on development mailing lists, at that time. With their agreement. I think I started my research back then due by my own question on: Will such a large disk actually work in AmigaOS?

It is all a long time ago…

Quoted 5 lines

If that still seems too risky, we can make the default behaviour to bail out once a potential overflow is detected, and allow the user to override that through a boot parameter. I'd leave that decision up for the code review on linux-block.

Quoted 7 lines

Two more comments: Linux uses 512 byte block sizes for the partition start and size calculations, so a change of the RDB blocksize to reduce the block counts stored in the RDB would still result in the same overflow. And amiga-fdisk is indeed utterly broken and needs updating (along with probably most legacy m68k partitioners). Adrian has advertised parted as replacement for the old tools - maybe this would make a nice test case for parted?

I always used AmigaOS based tools to partition in RDB format, cause I never trusted amiga-fdisk :).

Quoted 108 lines · jdow

Quoted 107 lines

If it is not backwards compatible I for one would refuse to use it. And if it still mattered that much to me I'd also generate a reasonable alternative. Modifying RDBs nay not be even an approximation of a good idea. You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it is for your personal use then you're entirely free to reject my advice and are probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for legacy disks only.

{^_^}

Quoted 93 lines · Michael Schmitz

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the issues Martin reported. Your patch addresses that by using the correct data type for the calculations (as do other partition parsers that may have to deal with large disks) and fixes Martin's bug, so appears to be the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations don't currently overflow is not expected to cause new issues, other than enabling use of disk and partitions larger than 2 TB (which may have ramifications with filesystems on these partitions). So comptibility is preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored in the RDB would enforce use of the same block size in filesystems. We'll have to rely on the filesystem tools to get that right, too. Linux AFFS does allow block sizes up to 4k (VFS limitation) so this should allow partitions larger than 2 TB to work already (but I suspect Al Viro may have found a few issues when he looked at the AFFS code so I won't say more). Anyway partitioning tools and filesystems are unrelated to the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on partitions larger than 2 TB, say so and I'll have the kernel print a warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a RDB image from a disk known to currently work under Linux (thanks Geert for the losetup hint). Can't do much more without procuring a working Amiga disk image to use with an emulator, sorry. The Amiga I plan to use for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:

Quoted 49 lines

As long as it preserves compatibility it should be OK, I suppose. Personally I'd make any partitioning tool front end gently force the
block size towards 8k as the disk size gets larger. The file systems
may also run into 2TB issues that are not obvious. An unused blocks
list will have to go beyond a uint32_t size, for example. But a block
list (OFS for sure, don't remember for the newer AFS) uses a tad under 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}

Just be sure you are aware of all the ramifications when you make a
change. I remember thinking about this for awhile and then determining I REALLY did not want to think about it as my brain was getting tied into a gordian knot.

{^_^}

Quoted 27 lines · Michael Schmitz

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)

sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)

Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>

Quoted 59 lines · wrote:

Quoted 59 lines

Quoted 50 lines

Quoted 50 lines

Quoted 49 lines

Quoted 49 lines

BTW - anybody who uses 512 byte blocks with an Amiga file system is
a famn
dool.

If memory serves the RDBs think in blocks rather than bytes so it
should
work up to 2 gigablocks whatever your block size is. 512 blocks is
2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
Go up to 4096 or 8192. The latter is 35 TB.

{^_^}

Quoted 19 lines · Martin Steigerwald

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 5 lines

test results at
https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep
which
I still have in my apartment.

Quoted 6 lines

So RDB support in Linux it remains broken for disks larger 2 TB,
unless
someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Michael.

Michael Schmitz - 26.06.18, 04:23:

Quoted 10 lines

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

Quoted 2 lines

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Well I pretty much believe that this was the standard block size of the tool I used in order to create the RDB. I think it has been Media

Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS- Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has been deprecated by AmigaOS upstream.

Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would have set it up differently.

Anyway: It works like this in AmigaOS 4 without any issues. With 512 Byte Blocks. I think it is good that Linux does not create data corruption when writing to disks that work just fine in AmigaOS. Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or Acube Sam machines may dual boot AmigaOS and Linux on their machines.

Thanks again for putting together a patch.

Thanks,

Michael.

Michael Schmitz - 26.06.18, 04:23:

Quoted 4 lines

Joanne,

Martin's boot log (including your patch) says:

Quoted 8 lines

Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]

Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512 byte blocks) and can be worked around by using a different block size.

Well I pretty much believe that this was the standard block size of the tool I used in order to create the RDB. I think it has been Media Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS- Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has been deprecated by AmigaOS upstream.

Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would have set it up differently.

Anyway: It works like this in AmigaOS 4 without any issues. With 512 Byte Blocks. I think it is good that Linux does not create data corruption when writing to disks that work just fine in AmigaOS. Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or Acube Sam machines may dual boot AmigaOS and Linux on their machines.

Thanks again for putting together a patch.

Thanks,

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 3 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 3 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

OK, I'll prepare a patch and submit it to linux-block for review. I'll have to refer to your testing back in 2012 since all I can test is whether the patch still allows partition tables on small disks to be recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI bridge to test this properly on).

Cheers,

Michael

Am 24.06.18 um 21:06 schrieb Martin Steigerwald:

Quoted 13 lines

Hi.

Michael Schmitz - 27.04.18, 04:11:

Quoted 4 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow�s fix.

Quoted 7 lines

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

OK, I'll prepare a patch and submit it to linux-block for review. I'll

have to refer to your testing back in 2012 since all I can test is whether the patch still allows partition tables on small disks to be recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI bridge to test this properly on).

Am 24.06.18 um 21:06 schrieb Martin Steigerwald:

Quoted 14 lines

Michael Schmitz - 27.04.18, 04:11:

Quoted 3 lines

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing the a patch from Joanne Dow´s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless someone else does.

On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald

Quoted 8 lines · <[email protected]> wrote:

You probably put your stick into a cave with ancient sleeping dragons

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

If there are bugs in the RDB parser that people run into, they should be fixed.
If there are limitations in the RDB format on large disks, that's still not a reason to move it to staging (hi msdos partitioning!).

Hi Martin,

CC jdow

Quoted 7 lines · Martin Steigerwald

You probably put your stick into a cave with ancient sleeping dragons :)

Added in linux-m68k mailing list, as they likely have an opinion on how to treat affs + RDB partition support. Also added in Jens Axboe about patching that RDB support broken with 2 TB or larger harddisks issue which had been in Linux kernel for 6 years while a patch exists that to my testing back then solves the issue.

While non-native Linux filesystem support (e.g. affs/isofs/...) could be handled by FUSE, moving RDB partition support to staging is not an option, as it is the only partitioning scheme that Amigas can boot from.

If there are bugs in the RDB parser that people run into, they should be fixed.
If there are limitations in the RDB format on large disks, that's still not a reason to move it to staging (hi msdos partitioning!).

Quoted 169 lines

Matthew Wilcox - 26.04.18, 04:57:

Quoted 12 lines · David Sterba

Quoted 10 lines

I had similar toughts some time ago while browsing the fs/ directory.
Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

Quoted 5 lines

Yeah, it's pretty sad how few commits some of these filesystems have had in recent years. One can argue that they're stable and don't need to be fixed because they aren't broken, but I find it hard to believe that any of them were better-implemented than ext2 which still sees regular bugfixes.

Regarding affs there is a severe issue which is not in affs itself but in the handling of Rigid Disk Block (RDB) partitions, the Amiga partitioning standard, which is far more advanced than MBR: It overruns for 2 TB or larger drives and then wraps over to the beginning of the drive – I bet you can imagine what happens if you write to an area larger than 2 TB. I learned this with an external 2TB RDB partitioned harddisk back then, which I used for Sam440ep (a kind of successor for old, classic Amiga hardware) backup + some Linux related stuff in another partition.

Joanne Dow, a developer who developed hdwrench.library which HDToolBox uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then, but never officially put it officially through upstreaming as I offered to make a good description and upstream it through Jens Axboe.

I may take this as a reason to… actually follow through this time, hopefully remembering all the details in order to provide a meaningful patch description – but I think mostly I can do just careful copy and paste. Even tough I believe Joanne Dow´s fix only fixed my bug report 43511, but not 43511 which is more about a safeguarding issue in case of future overflows, I still think it would be good to go in in case affs + RDB stays in their current places.

However, in case you move affs to staging, I may be less motivated to do so, but then I suggest you also move RDB partitioning support to staging, cause this is the one that is known to be dangerously badly for 2 TB or larger disks. And yeah, I admit I did not follow through with having that patch upstreamed. Probably I did not want to be responsible in case my description would not have been absolutely accurate or the patch breaks something else. I do not have that 2 TB drive anymore and don´t feel like setting one up in a suitable way in order to go about this patch, but my testing back then was quite elaborate and I still feel pretty confident about it.

I totally get your motivation, but I would find it somewhat sad to see the filesystems you mentioned go into staging. However, as I just shown clearly, for the user it may be better, cause there may be unfixed dangerous bugs. FUSE might be an interesting approach, but I bet it will not solve the maintenance issue. If there is no one maintaining it in the kernel, I think its unlikely to find someone adapting it to be a FUSE filesystem and maintaining it. And then I am not aware of FUSE based partitioning support. (And I think think ideally we´d had a microkernel and run all filesystems in userspace processes with a defined set of privileges, but that is simply not Linux as it is.)

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://lkml.org/lkml/2012/6/17/6

Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
https://bugzilla.kernel.org/show_bug.cgi?id=43511

I forward the relevant mail of Joanne, in

https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7

I even have the patch in diff format. And I just checked, the issue is still unpatched as of 4.16.3.

---------- Weitergeleitete Nachricht ----------

Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1
Datum: Montag, 18. Juni 2012, 03:28:48 CEST
Von: jdow <[email protected]>
An: Martin Steigerwald <[email protected]>
Kopie: Geert Uytterhoeven <[email protected]>, linux- [email protected], Jens Axboe <[email protected]>, linux- [email protected]

Quoted 17 lines · Martin Steigerwald

Am Sonntag, 17. Juni 2012 schrieb jdow:

Quoted 2 lines · Geert Uytterhoeven

Quoted 1 line

On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald

Quoted 13 lines · <[email protected]> wrote:

Quoted 13 lines

Quoted 13 lines

Am Sonntag, 17. Juni 2012 schrieb jdow:

| JXFS 64 bit file system
|
| With AmigaOS 4.x a new file system has been introduced called
| JXFS. It is a totally new 64 bit file system that supports
| partitions up to 16 TB in size. It is a modern journalling file
| system, which means that it reduces data loss if data writes to
| the disk are interrupted. It is the fastest and most reliable
| file system ever created for AmigaOS.

http://www.amigaos.net/content/1/features

Well I asked AmigaOS 4 developers about this issue as well. Lets

see

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

what they say about 2 TB limits.

16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?

block/partitions/amiga.c reads the block size from RigidDiskBlock.rdb_BlockBytes,
but after conversion to 512-byte blocks, all further calculations

are

Quoted 10 lines

Quoted 5 lines

Quoted 5 lines

done on "int", so it will overflow for disks larger than 2 TiB.

Note that in your profile-binary.img, the field is 0x200, i.e. 512 bytes per block,
so I'll have to get a deeper look into your RDB first...

[…]

Quoted 2 lines

Note that the work I did on the Linux RDB code eons ago took full cognizance of the physical and virtual block sizes.

[…]

Quoted 1 line

I've asked Martin for a digital copy of his RDBs and what he thinks

the

Quoted 8 lines

Quoted 2 lines

partition(s) should look like. I should also be told whether the disk is supposed to be solely Amiga OSs or not. I gather it's not.

Its all in the bug report. profile-binary.img should be it.

Its small so I attach it.

Meanwhile I try to get the currently supported maximum disk size out

from

Quoted 1 line

the OS 4 developers. Maybe JXFS is playing tricks that other

filesystems do

Quoted 3 lines

not play or simply has a different implementation.

Thanks,

I've sent Jens a proposed fix changing these lines in amiga.c. ===8<--- was

          struct PartitionBlock *pb;
          int start_sect, nr_sects, blk, part, res = 0;
          int blksize = 1;        /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
          struct PartitionBlock *pb;
          sector_t start_sect, nr_sects;
          int blk, part, res = 0;
          int blksize = 1;        /* Multiplier for disk block size */

===8<---

(I'm working without proper tools at the moment or I'd make a real diff. Sorry.)

This fix may not be complete. The RDBs contain provisions for describing the physical disk block size and how many physical blocks are used in a file system's logical blocks. This MAY not work quite right large physical
block sizes.

Quoted 16 lines

{^_^} Joanne Dow
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-------------------------------------------------------------
[…]

Quoted 7 lines

As long as there's someone who can be pestered with bugs, I don't see the need to push their baby out of the tree. I'm a bit more nervous about hfsplus; if it has users and no maintainer, those users are at risk.

Perhaps we could advertise on Craigslist or something ... maintainer wanted for LTR.

Gr{oetje,eeting}s,

Geert

And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS as well as Linux OS.

Quoted 8 lines

And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS as well as Linux OS.

I assume removing the feature for AmigaOS isn't really possible since we don't have the source code for that, do we?

Quoted 4 lines · jdow

And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS as well as Linux OS.

I assume removing the feature for AmigaOS isn't really possible since we don't have the source code for that, do we?

Also, if I remember correctly, Mac partitions can store filesystem drivers as well and its actually a feature being used in MacOS. parted received a patch some time ago to fix the correct handling for storing the filesystem driver in the partition table.

I would be generally against removing these features as I don't think the security risk is relevant for the majority of users. The Amiga is a hobbyist machine these days and AmigaOS has certainly way more on than way to be compromised through vulnerabilities.

Adrian

Quoted 17 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We have no maintainer for hfs, and no likely prospect of anyone stepping up soon to become hfs maintainer. I think it's irresponsible of us to present unmaintained code on an equal basis with filesystems under active maintenance like ext2.

I therefore propose we move the following filesystems which are explicitly listed as Orphaned to staging:

affs - Amiga filesystem.
efs - old SGI filesystem predating XFS, used on CDs for a while. hfs - Mac filesystem.
hfsplus - Mac filesystem.

Quoted 6 lines

I further propose we move the following filesystems which have no entry in MAINTAINERS to staging:

adfs - Acorn filesystem from the 1990s.
minix
qnx6

I had similar toughts some time ago while browsing the fs/ directory. Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

Quoted 29 lines · David Sterba

Quoted 17 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We have no maintainer for hfs, and no likely prospect of anyone stepping up soon to become hfs maintainer. I think it's irresponsible of us to present unmaintained code on an equal basis with filesystems under active maintenance like ext2.

I therefore propose we move the following filesystems which are explicitly listed as Orphaned to staging:

affs - Amiga filesystem.
efs - old SGI filesystem predating XFS, used on CDs for a while. hfs - Mac filesystem.
hfsplus - Mac filesystem.

Quoted 6 lines

I further propose we move the following filesystems which have no entry in MAINTAINERS to staging:

adfs - Acorn filesystem from the 1990s.
minix
qnx6

I had similar toughts some time ago while browsing the fs/ directory. Access to the filesystem images can be reimplemented in FUSE, but other than that, I don't think the in-kernel code would be missed.

Quoted 6 lines

It's hard to know how many users are there. I was curious eg. about bfs, befs, coda or feevxfs, looked at the last commits and searched around web if there are any mentions or user community. But as long as there's somebody listed in MAINTAINERS, the above are not candidates for moving to staging or deletion.

I think the presence of maintainers entry is necessary but insufficient. What if the maintainer has long lost interest but just didn't bother updating the file. In the very least maintainers should be pinged and asked if they are still maintaining the respective piece of code.

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

Quoted 4 lines

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.

Pavel Machek - 26.04.18, 08:11:

Quoted 6 lines · Matthew Wilcox

Quoted 4 lines

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

Quoted 7 lines

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.

Good point.

Moving things in and out to some "unmaintained" directory… I am not sure about that either. I tend to think that moving code around does not solve the underlying issue.

Which, according to what I got from Matthew, was that distributors enable just about every filesystem they can enable which lead to hfs being used for automounting an USB stick formatted with it.

Quoted 3 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.

Quoted 3 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.

Quoted 11 lines · Pavel Machek

Quoted 3 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Quoted 1 line

Staging is for ugly code, not for code that needs new maintainter.

Staging is used for getting code _out_ of the kernel tree as well as _in_. We use it all the time to move code there, see if anyone shows up in 6-8 months to say "I will fix this!", and if not, we delete it.

Look at what just happened to IRDA in the 4.17-rc1 release as an example of this.

Quoted 14 lines · On Sunday 29 April 2018 14:07:05 Greg KH wrote:

Quoted 12 lines · Pavel Machek

Quoted 3 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.

Quoted 6 lines

Staging is used for getting code _out_ of the kernel tree as well as _in_. We use it all the time to move code there, see if anyone shows up in 6-8 months to say "I will fix this!", and if not, we delete it.

Look at what just happened to IRDA in the 4.17-rc1 release as an example of this.

Really a "great" example of deleting working code :(

Quoted 15 lines · Ondrej Zary

Quoted 14 lines · On Sunday 29 April 2018 14:07:05 Greg KH wrote:

Quoted 12 lines · Pavel Machek

Quoted 3 lines · Matthew Wilcox

Recently ncpfs got moved to staging. Also recently, we had some fuzzer developers report bugs in hfs, which they deem a security hole because Ubuntu attempts to automount an inserted USB device as hfs.

We promise "no-regressions" for code in main repository, no such promise for staging. We have quite a lot of code without maintainer.

Moving code to staging means it will get broken -- staging was not designed for this. I believe moving anything there is bad idea.

Staging is for ugly code, not for code that needs new maintainter.

Quoted 8 lines

Quoted 6 lines

Staging is used for getting code _out_ of the kernel tree as well as _in_. We use it all the time to move code there, see if anyone shows up in 6-8 months to say "I will fix this!", and if not, we delete it.

Look at what just happened to IRDA in the 4.17-rc1 release as an example of this.

Really a "great" example of deleting working code :(

What do you mean? The irda code was broken and not working at all. There were loads of bug reports about it for years, with no developers or maintainers willing to do the work on it to get it to actually work again.

If someone does want to step up and do it, great! It's a simple revert of two git commits and they are back in business.

Dropping code from the tree is not like it is gone for forever. If someone wants to pick it up, it is trivial to do so.