Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

Hi Jens, hi Linux m68k developers,

I reported that as

https://bugzilla.kernel.org/show_bug.cgi?id=43511

I will attach there some more debug data like binary copy of RDB and such.

But maybe its easier to discuss here.

I am not sure whether its an issue with Linux or an issue with the RDB format and disks with big sizes. But AFAIK RDB format is capable of handling 2 TB disks.

With my 2 TB mixed Amiga/Linux backup disk, which I partioned under AmigaOS 4.0/1 with Media Toolbox I get the following in Linux:

Jun 17 07:28:14 merkaba kernel: [30852.968978] sata_sil24 0000:05:00.0: 
enabling device (0000 -> 0003)
Jun 17 07:28:14 merkaba kernel: [30852.969401] scsi9 : sata_sil24
Jun 17 07:28:14 merkaba kernel: [30852.969533] ata10: SATA max UDMA/100 
host m128@0xf1c02000 port 0xf1c00000 irq 19
Jun 17 07:28:16 merkaba kernel: [30855.163712] ata10: SATA link up 3.0 
Gbps (SStatus 123 SControl 0)
Jun 17 07:28:16 merkaba kernel: [30855.165014] ata10.00: ATA-8: Hitachi 
HDS5C3020ALA632, ML6OA580, max UDMA/133
Jun 17 07:28:16 merkaba kernel: [30855.165017] ata10.00: 3907029168 
sectors, multi 16: LBA48 NCQ (depth 31/32)
Jun 17 07:28:16 merkaba kernel: [30855.166378] ata10.00: configured for 
UDMA/100
Jun 17 07:28:16 merkaba kernel: [30855.166477] scsi 9:0:0:0: Direct-Access     
ATA      Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5
Jun 17 07:28:16 merkaba kernel: [30855.166653] sd 9:0:0:0: [sdb] 
3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Jun 17 07:28:16 merkaba kernel: [30855.166699] sd 9:0:0:0: [sdb] Write 
Protect is off
Jun 17 07:28:16 merkaba kernel: [30855.166702] sd 9:0:0:0: [sdb] Mode 
Sense: 00 3a 00 00
Jun 17 07:28:16 merkaba kernel: [30855.166726] sd 9:0:0:0: [sdb] Write 
cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 17 07:28:16 merkaba kernel: [30855.200936]  sdb: RDSK (512) sdb1 
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4)
Jun 17 07:28:16 merkaba kernel: [30855.200943] sdb: p1 size 
18446744072560312368 extends beyond EOD, enabling native capacity
Jun 17 07:28:16 merkaba kernel: [30855.201344]  sdb: RDSK (512) sdb1 
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4)
Jun 17 07:28:16 merkaba kernel: [30855.201347] sdb: p1 size 
18446744072560312368 extends beyond EOD, truncated
Jun 17 07:28:16 merkaba kernel: [30855.201398] sdb: p2 start 
18446744072560314432 is beyond EOD, truncated
Jun 17 07:28:16 merkaba kernel: [30855.201400] sdb: p3 start 
18446744073189460080 is beyond EOD, truncated
Jun 17 07:28:16 merkaba kernel: [30855.201570] sd 9:0:0:0: [sdb] Attached 

SCSI disk

The first partition seems to be way to big:

merkaba:~#1> amiga-fdisk -l /dev/sdc
Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder

Device Boot Mount Begin End Size Pri BBlks System
/dev/sdc1 * 43 65536043 1572864024 0 0 Linux
native
/dev/sdc2 * 65536044 78643244 314572824 0 0
[unknown]
/dev/sdc3 * 78643245 81396440 66076704 0 0 Amiga
FFS Int.

(sdc2 is JXFS, a new filesystem in AmigaOS 4 that is not known to Linux yet)

In cat /proc/partitions I get:

merkaba:~> cat /proc/partitions
major minor #blocks name

[…]
8 16 1953514584 sdb
8 17 1953513552 sdb1
merkaba:~>

Thus the Debian Linux Kernel 3.4.1 I am using here, truncates the first oversized partition and has no space for the other, too, which therefore are inaccessible under Linux.

I didn´t notice this initially, but it happened from the beginning, I have an old amiga-fdisk listing that is exactly the same.

Amiga Mediatoolbox has a different oppinion on the partition layout:

LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to 81396440 cyl, didn´t note the size here, but as far as I remember was okay as well

But it seems to be confused about the whole size of the disk as well:

Logical sizes:

Blocks per cylinder: 48
Cylinders: 81396441
Sectors: -397938128
Blocksize: 512

The sectors seem overflowed.

So it might be a problem with RDB and 2TB disks and nothing to do with Linux. But still on AmigaOS 4.1 I can access the two Amiga partitions after the Linux partition.

I have another 500GB disk for backup up my Sam440ep AmigaOS 4.1 "Amiga", I plan to repartition the 2 TB disk as GPT anyway, but since MediaToolBox in AmigaOS 4.1 has a different meaning about the partioning and this can cause serious data loss, I think its good to look at it.

I had a BTRFS filesystem that had some checksum errors. Maybe thats somehow related to this issue and AmigaOS and/or Linux overwrote something it shouldn´t have touched.

I will report a bug with AmigaOS 4.1 developers as well to get more details.

merkaba:~> cat /proc/version
Linux version 3.4-trunk-amd64 (Debian 3.4.1-1~experimental.1) ([email protected]) (gcc version 4.6.3 (Debian 4.6.3-1) ) #1 SMP
Wed Jun 6 10:34:53 CEST 2012

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Am Sonntag, 17. Juni 2012 schrieb Martin Steigerwald:

I think this one is pretty serious:

merkaba:~> pvdisplay /dev/sdb1
--- Physical volume ---

  PV Name               /dev/sdb1
  VG Name               steigerwald
  PV Size               1,82 TiB / not usable 4,08 MiB
  Allocatable           yes 
  PE Size               4,00 MiB
  Total PE              476931
  Free PE               105731
  Allocated PE          371200
  PV UUID               ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp

merkaba:~> vgdisplay steigerwald
--- Volume group ---
VG Name steigerwald
System ID

  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                5
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1,82 TiB
  PE Size               4,00 MiB
  Total PE              476931
  Alloc PE / Size       371200 / 1,42 TiB
  Free  PE / Size       105731 / 413,01 GiB
  VG UUID               uhjjE1-0yrD-Ch1A-d9qL-P5jY-UDXE-io4bpi

with

merkaba:~#1> amiga-fdisk -l /dev/sdc
Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder

Device Boot Mount Begin End Size Pri BBlks System

/dev/sdc1       *        43   65536043   1572864024     0      0  Linux native
/dev/sdc2       *    65536044   78643244   314572824     0      0  [unknown]
/dev/sdc3       *    78643245   81396440   66076704     0      0  Amiga FFS Int.

Due to truncating oversized partition instead of refusing to use them the PV and consequently VG span the whole disk instead of leaving space for the >350GB in the two other partitions:

merkaba:~> cat /proc/partitions
major minor #blocks name

[…]
8 16 1953514584 sdb
8 17 1953513552 sdb1

(now sdb instead of sdc back when I run amiga-fdisk)

This is just asking for trouble. Thus:

Bug 43521 - Amiga RDB partitions: truncates miscaluted partitions size instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

My suggestion is to bail out if a partition size looks insane and impossible. Having to manually fix it up to access the data on disk still looks better to me than potentially overwriting lots of data.

Until today I never actually did an Amiga backup to the disk, I just created the two Amiga partitions and copied some screenshots of them but that might be enough to explain the checksum errors I found in a BTRFS backup partition.

I am now checking my Linux backups. They might be save, cause not all space on the volume group is used.

See:

kernel got struck while scrubbing BTRFS with node- and leafsize 32768
http://www.mail-archive.com/[email protected]/msg17108.html

s/struck/stuck

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

I'll have to make time to look it over, considering I am more or less the mommy of the RDBs. Now, 2TB may be a tad beyond the abilities of the RDBs to express. Memory insists that the RDBs worked on either BYTE or block counts. Your seeing a problem at 2TB suggests it really stored bit counts. The numbers are supposed to be unsigned. And whether the RDBs store values in blocks or BYTEs is immaterial when you get down to it. (The RDBs do store the disk's actual and virtual block sizes. The latter is what the filesystem uses.)

However, since C programs seek plus and minus in BYTEs you are stuck with 31 bits worth of disk as your limit. That matches nicely with your 31 bits. The RDBs should still be useful if you make sure to plant them on block 1 or higher leaving block zero for other OSs and you only use the first 2 TB for the Amiga OS files. (RDPrep and the HDWrench library both exhibited a strong preference for starting the RDB blocks on block 3 of the disk.)

{^_^} Joanne Dow, yeah, that antique broad.

Am Sonntag, 17. Juni 2012 schrieb jdow:

Hmmm, I thought any 2 TB limit was gone meanwhile, but then your argumentation has some merit.

Some hints that it might be gone nonetheless:

| JXFS 64 bit file system
|
| With AmigaOS 4.x a new file system has been introduced called JXFS. It is 
| a totally new 64 bit file system that supports partitions up to 16 TB in 
| size. It is a modern journalling file system, which means that it reduces 
| data loss if data writes to the disk are interrupted. It is the fastest 
| and most reliable file system ever created for AmigaOS.

http://www.amigaos.net/content/1/features

Well I asked AmigaOS 4 developers about this issue as well. Lets see what they say about 2 TB limits.

But nonetheless, I think, Linux should refuse to use a partition that is clearly out of bound.

Scrubbing the one backup BTRFS that has 360 GiB of data was okay. Also the fsck´s went well. Well the volume group is not completely full so some space on the end of the disk is not yet used by it.

I am recreating all my backups on BTRFS volumes now - except maybe the one thats on the BTRFS volume already and scrubs okay.

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?

block/partitions/amiga.c reads the block size from RigidDiskBlock.rdb_BlockBytes,
but after conversion to 512-byte blocks, all further calculations are done on "int", so it will overflow for disks larger than 2 TiB.

Note that in your profile-binary.img, the field is 0x200, i.e. 512 bytes per block,
so I'll have to get a deeper look into your RDB first...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds

Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:

I hope to get anything back from the AmigaOS 4 developers.

Okay, thanks!

I did not get any information regarding the current size limit yet.

Strangely on AmigaOS 4.1 all values seem to be fine except for the total sectors value.

And on Linux begin and end cylinders are correct, only size is off:

merkaba:~> amiga-fdisk -l /dev/sdb
Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder

Device Boot Mount Begin End Size Pri BBlks System

/dev/sdb1       *        43   65536043   1572864024     0      0  Linux native
/dev/sdb2       *    65536044   78643244   314572824     0      0  [unknown]
/dev/sdb3       *    78643245   81396440   66076704     0      0  Amiga FFS Int.

But not only from the first, also of the second and third one it seems.

65536043 - 43 = 65536000

78643244 - 65536044 = 13107200

81396440 - 78643245 = 2753195

I would think that if there is a overflow it would already be in the cylinder numbers.

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Ah, you DO know that "cylinders, surfaces, and tracks" are polite fictions in AmigaOS, don't you? Start and End blocks are all that matter on a real Amiga. The fictions arose because at first it was thought they could be used to optimize disk accesses. Once drives were notched these values became meaningless. So they're created on the fly picking values out of the nose or something. (RDPrep tries to find reasonable size factors for the total block counts.)

Are you sure "amiga-fdisk" is not broken?

So the size in bytes is 24 times the byte offset of the start of the next partition. Fascinating. Let's see. You are working in 512 byte blocks it looks like. With RDBs in blocks that means you can get up to 1099511627776 bytes, 2147483648 blocks, or 44739242. So you are already WAY over what can be expressed in the 32 bit values in the RDBs. So the software that prepared your partitioning needs some repair work of some sort or something other than traditional Amiga FFS format disks.

The first thing on the agenda is "fixing" the partitioning software. Is the version of Amiga FFS you are using cognizant of 64 bit values? If not you will have to go to block sizes larger than 512 bytes. It looks like 1k is suitable for this instance. Given the way Amiga FFS stores data on the disk I'd go for 4k or 8k block sizes unless you have lots of very small files.

The RDB reading software may not be at fault. I'll have to check. Your partitioning tool let you down. It looks like you've overflowed the 31 (useful) bit values in your RDBs. So not much can recover from this. Play with signed arithmetic and treat the final value as unsigned and see if you get the size noted.

{^_^}

Am Sonntag, 17. Juni 2012 schrieb jdow:

I know there are pure fiction on Linux as well. As in any other modern operating system.

Actually Media Toolbox shows both. Physical and logical sizes. See the screenshots I attached to the bug report.

Not at all, but there is also the syslog.

Is there still something to fix in there? Just still catching up the mail exchange. From what I could see on AmigaOS 4.1 all seemed well. I already reported the negative sector count value.

Anyway, if there is an issue left we can discuss this privately. This would have nothing to do with Linux and I can make sure that current AmigaOS developers hear about your oppinion.

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

On 2012/06/17 14:06, Martin Steigerwald wrote:
...

I have a usage note for you and the person who did that partitioning. If you start the partition at block 1 instead of block zero it probably can coexist nicely with normal fdisk partitioning as well. (It used to be able to since x86 partition tables didn't bother with blocks on the first "cylinder" other than block zero.) I designed RDPrepX and HDWrench to take advantage of that. (And I used it myself. Erm I used a LOT of abominations such as safely nested partitions.)

{^_^}

Note that the work I did on the Linux RDB code eons ago took full cognizance of the physical and virtual block sizes.

I still have, I believe, a working Fuji Magneto Optical drive with 2k sectors. I used that for ages for moving data from systems at two different houses as I moved back and forth. I worked on the Linux RDB code because I wanted to be able to read those disks. I've been vaguely wondering what happened to the code in the latest builds. Now I guess I will find out.

If RDBs are going to remain backwards compatible and AmigaOS disk usage is to remain sensible larger logical blocks, at the very least, are needed. Since both values (should) exist within the RDBs the partitions that are described in the RDBs can be managed by reading the logical block size and multiplying it by the ending block number storing as 64 bits. It sounds like this is not being done correctly. I may need some guidance to see about where to put the 64 bit byte position of the starting and ending blocks.

I've asked Martin for a digital copy of his RDBs and what he thinks the partition(s) should look like. I should also be told whether the disk is supposed to be solely Amiga OSs or not. I gather it's not.

(And for God's sake do NOT implement the two virus storage tools within the RDBs, the RDB encapsulated filesystems and the RDB encapsulated driver code. I worried about that potential since RDBs were first introduced. Oddly I never heard of them being used. So I kept quiet about it. These days those facilities could be extended to use signing and validation certificates, I suppose.)

{^_^}

Hi Joanne,

His RDB is attached to https://bugzilla.kernel.org/show_bug.cgi?id=43521

JFYI, I used the former to store my MultiUser File System ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds

I want the raw binary on the blocks 0 through the end of the RDBs or 128 blocks, which ever comes first. I don't see that there. I just see the amiga-fdisk printout of it. With the actual binary I can parse it myself and see what comes out.

(As a side note - RDBs work with large sector sizes. They pretty much always have.)

...

{^_^}

Am Sonntag, 17. Juni 2012 schrieb jdow:

[…]

[…]

Its all in the bug report. profile-binary.img should be it.

Its small so I attach it.

Meanwhile I try to get the currently supported maximum disk size out from the OS 4 developers. Maybe JXFS is playing tricks that other filesystems do not play or simply has a different implementation.

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Ah it was on the report to which your bugzilla entry I had referred to. OK, it looks like it's not been overwritten - one of the usual overflow symptoms. Now I have to hook it up with my handy dandy translator and see what that says. I couldn't find it on 43521.

{^_^}

I've sent Jens a proposed fix changing these lines in amiga.c. ===8<--- was

        struct PartitionBlock *pb;
        int start_sect, nr_sects, blk, part, res = 0;
        int blksize = 1;        /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
        struct PartitionBlock *pb;
        sector_t start_sect, nr_sects;
        int blk, part, res = 0;
        int blksize = 1;        /* Multiplier for disk block size */

===8<---

(I'm working without proper tools at the moment or I'd make a real diff. Sorry.)

This fix may not be complete. The RDBs contain provisions for describing the physical disk block size and how many physical blocks are used in a file system's logical blocks. This MAY not work quite right large physical block sizes.

{^_^} Joanne Dow

Am Montag, 18. Juni 2012 schrieb jdow:

Way better:

dmesg:

Jun 19 21:19:09 merkaba kernel: [ 7891.819552] ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
Jun 19 21:19:09 merkaba kernel: [ 7891.821306] ata8.00: ATA-8: Hitachi HDS5C3020ALA632, ML6OA580, max UDMA/133
Jun 19 21:19:09 merkaba kernel: [ 7891.821315] ata8.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32)
Jun 19 21:19:09 merkaba kernel: [ 7891.823252] ata8.00: configured for UDMA/100
Jun 19 21:19:09 merkaba kernel: [ 7891.823589] scsi 7:0:0:0: Direct-Access     ATA      Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5
Jun 19 21:19:09 merkaba kernel: [ 7891.824126] sd 7:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Jun 19 21:19:09 merkaba kernel: [ 7891.824440] sd 7:0:0:0: [sdb] Write Protect is off
Jun 19 21:19:09 merkaba kernel: [ 7891.824452] sd 7:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Jun 19 21:19:09 merkaba kernel: [ 7891.824531] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1 (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)
(res 2 spb 4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb] Attached SCSI disk

cat /proc/partitions:
major minor #blocks name
[…]

   8       16 1953514584 sdb
   8       17 1572864024 sdb1
   8       18  314572824 sdb2
   8       19   66076704 sdb3

1572864024 + 314572824 + 66076704 = 1953513552 which is below the complete size of the disk.

Partition start analysis:

merkaba:~> file -sk /dev/sdb1
/dev/sdb1: sticky LVM2 PV (Linux Logical Volume Manager), UUID: ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp, size: 2000397877248 merkaba:~> file -sk /dev/sdb2
/dev/sdb2: sticky data
merkaba:~> file -sk /dev/sdb3
/dev/sdb3: sticky Amiga Inter FFS disk

merkaba:~> hd /dev/sdb2 | head

00000000  4a 58 46 04 11 1a 0f 0c  00 00 00 00 00 00 00 00  |JXF.............|
00000010  00 00 00 00 00 11 00 00  3e db 3d 54 40 00 00 00  |........>.=T@...|
00000020  00 00 00 00 00 00 00 00  00 00 01 77 00 10 80 00  |...........w....|
00000030  00 00 01 c2 00 10 e0 00  25 80 00 30 00 00 02 00  |........%..0....|
00000040  00 00 00 30 00 00 00 00  00 00 00 00 00 00 00 00  |...0............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 08 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*
00000200 54 52 4f 4b ab ad b0 b1 00 00 00 01 00 00 00 00 |TROK............|

I don´t know the on-disk format for JXFS, but this could be it. JFX/04 pretty much looks like the DOS type for JXFS. Yes, thats it. JXF/04 is the DOS type in use for JXFS as I verified by looking at a JFXS partition with Media Toolbox.

amiga-fdisk looks the same as before:

merkaba:~> amiga-fdisk -l /dev/sdb
Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder

Device Boot Mount Begin End Size Pri BBlks System

/dev/sdb1       *        43   65536043   1572864024     0      0  Linux native
/dev/sdb2       *    65536044   78643244   314572824     0      0  [unknown]
/dev/sdb3       *    78643245   81396440   66076704     0      0  Amiga FFS Int.

But obviously its right anyway: It seems to display the size in blocks, not in cylinders.

Media Toolbox says:

LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to 81396440 cyl, didn´t note the size here, but as far as I remember was okay as well

with

Blocks per cylinder: 48
Cylinders: 81396441

So thats:

65536001 * 48 / 2 = 1572864024

13107201 * 48 / 2 = 314572824

( 81396440 - 78643245 ) * 48 / 2 = 66076680

(I verified from a windowshot I made that 81396440 - 78643245 = 2753195 is indeed displayed by Media Toolbox as size of the partition)

So this is looking pretty good.

Many thanks, Joanne for your detective work and the fix.

Tested with:

martin@merkaba:~> cat /proc/version
Linux version 3.5.0-rc3-fix-bug-43511+ (martin@merkaba) (gcc version 4.7.0 (Debian 4.7.1-1) ) #1 SMP Tue Jun 19 14:31:56 CEST 2012

Patch from Joanne in diff format:

martin@merkaba:~/Computer/Merkaba/Kernel/linux-2.6> git diff HEAD~ | cat

index 70cbf44..42d36f9 100644
@@ -29,7 +29,8 @@ int amiga_partition(struct parsed_partitions *state)
        unsigned char *data;
        struct RigidDiskBlock *rdb;
        struct PartitionBlock *pb;
-       int start_sect, nr_sects, blk, part, res = 0;
+       sector_t start_sect, nr_sects;
+       int blk, part, res = 0;
        int blksize = 1;        /* Multiplier for disk block size */
        int slot = 1;
        char b[BDEVNAME_SIZE];

Jens, please take that patch. If you want I can prepare it further with above testing report and some explaination as changelog and a From Joanne Dow line ;). But it might be easier if you just take it as is and attribute it to her. Feel free to use as much of my test report as you want for in commit message. But I will copy this into the bug report anyway.

If you taken it, please tell me the commit id. I think this should go to stable kernel since its a potential data loss issue and an isolated overflow fix.

Only LVM is unhappy now, but thats to be expected since /dev/sdb1 that it used is now smaller than the whole disk as it was before with the overflowing and truncating Amiga RDB code without Joanne´s fix.

merkaba:~> vgscan
Reading all physical volumes. This may take a while...
/dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig
WARNING: Inconsistent metadata found for VG steigerwald - updating to use version 7 /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig Automatic metadata correction failed
Recovery of volume group "steigerwald" failed.
Found volume group "merkaba" using metadata type lvm2

So I will now format the disk as GPT and use it only for Linux for now, as I now have a different backup disk for the Amiga anyway.

So I have to recreate backups once again. Oh well, I possibly could still boot an older kernel to access the LVM again.

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:

Meanwhile I got a response from a AmigaOS 4 developer.

I documented the stuff as I understood it in the AmigaOS wiki:

| Disk size
|
| The RDB has a quite high limit on the maximum device size, but note that 
| currently each filesystem interprets the partition layout by itself.

| The raw, theoretical limit on the maximum device capacity is about 2^105 
| bytes:
|
| 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512 
| bytes/sector for the HD size in struct RigidDiskBlock
|
| It's even much more if the sector size (rdb_BlockBytes and de_SizeBlock) 
| is larger than 512 bytes, but AmigaOS 4.1 doesn't support anything but 
| 512 bytes/sector HDs yet.
|
| Partition size
|
| For the partitions the maximum size is:
| 32 bit (de_HighCyl + 1 - de_LowCyl) (to get the partition size) * 32 bit 
| de_Surfaces * 32 bit de_SectorsPerTrack * 512 bytes/sector in struct 
| DosEnvec (=pb_Environment[]) in struct PartitionBlock.
|
| That's from the physical drive part, the actual disk size limit for the 
| partitions may be much smaller depending on the partitioning software, 
| if it's only using the logical sizes instead, which is likely the case, 
| it's only 8 ZiB with 512 bytes/sector: 32 bit rdb_HiCylinder * 32 bit 
| rdb_CylBlocks * 512 bytes/sector = 2^73 bytes. For using the logical 
| sizes using simple uint64 calculations (with some overflow checks) should 
| be enough, for more a math library with support for larger integers 
| needs to be used which probably no partitioning software does.
|
| But note: Nothing in struct RigiDiskBlock is used by the file systems for 
| mounting the partitions, they only get the information from the struct 
| PartitionBlock blocks, it's only a problem for the partitioning software 
| creating the partitions correctly - as soon as there are HDs larger than 
| 8 ZB while still using 512 bytes/sector if that ever happens.

http://wiki.amigaos.net/index.php/RDB

Please note that the documentation there might be updated or corrected in the future. But thats the current state.

I am a bit overwhelmed by Joanne´s analysis. I didn´t yet take time to completely read and try to understand it. I first wanted to get this information out.

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

He's in the right ballpark but he missed something in the RDISK block:

    __u32   rdb_LoCylinder;	// 88	0x2b		Ayup - checks
    __u32   rdb_HiCylinder;	// 8C   0x04da02d8	Whole disk used.
    __u32   rdb_CylBlocks;	// 90	0x30		checks

So he has two 32 bit unsigned integers not three to multiply together. If he ignores that detail he can, indeed, go out to three ULONGS for the total disk logical block count. Then he has two more ULONGs for physical block size and number of physical blocks per filesystem blocks. So the size could, in theory, go past 2^100 bytes.

The filesystem is still likely limited to 2 TB or so unless there is a 64 bit version now.

When you can please recompile a kernel with that one change to it. It MIGHT turn the trick. It changes an int to a sector_t (unsigned long long) so the math does not go funkity going into the "put_partition" function, which has disk blocks as sector_t values. (I think, but can't be sure, that the put_partition call used int when I "committed" that sin way back when. I've not yet compared that code with the code I submitted many years ago to see if all the right stuff is still there.)

{^_^}