aoe fails on sparc64
Subsystems (2)
Hello,
Using aoe on a sparc64 system gives strange results:
sunny:/dev/etherd# echo >discover sunny:/dev/etherd# mke2fs e0.0 mke2fs 1.37 (21-Mar-2005) mke2fs: File too large while trying to determine filesystem size sunny:/dev/etherd# blockdev --getsz e0.0 -4503599627370496
The log says:
Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6> etherd/e0.0: unknown partition table Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has
67553994410557440
sectors
The system is an Sun Ultra 5, running 2.6.12.5/sparc64 compiled with gcc-3.4.2. e0.0 is exported on a x86 system using vblade-5, and has a size of 30 MB.
Regards,
Jim
Quoted 4 lines · Jim MacBaine
Hello,
Using aoe on a sparc64 system gives strange results:
...
Quoted 7 lines
The log says:
Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6> etherd/e0.0: unknown partition table Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has67553994410557440
sectors
OK. 67553994410557440 is 61440 byte swapped in 64 bits, and 30MB is 61440 sectors, so this should be a simple byte order fix.
Quoted 3 lines
The system is an Sun Ultra 5, running 2.6.12.5/sparc64 compiled with gcc-3.4.2. e0.0 is exported on a x86 system using vblade-5, and has a size of 30 MB.
Thanks for the report.
Signature · 1 line
Ed L Cashin <[email protected]>
From: Ed L Cashin <[email protected]>
Date: Wed, 31 Aug 2005 11:50:55 -0400
Quoted 10 lines
Quoted 5 lines · Jim MacBaine
Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6> etherd/e0.0: unknown partition table Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has67553994410557440
sectorsOK. 67553994410557440 is 61440 byte swapped in 64 bits, and 30MB is 61440 sectors, so this should be a simple byte order fix.
More strangely, the upper and lower 32-bit words are swapped. The bytes within each 32-bit word are swapped correctly.
So the calculation maybe should be something like:
__le32 *p = (__le32 *) &id[100 << 1]; u32 high32 = le32_to_cpup(p); u32 low32 = le32_to_cpup(p + 1); ssize = (((u64)high32 << 32) | (u64) low32);
But that doesn't make any sense, and even ide_fix_driveid() in drivers/ide/ide-iops.c does a le64_to_cpu() for this value:
id->lba_capacity_2 = __le64_to_cpu(id->lba_capacity_2);
I wonder if this is some artifact of how AOE devices encode this field when sending it to the client.
Quoted 1 line · David S. Miller
From: Ed L Cashin <[email protected]>
...
Quoted 14 lines
Quoted 2 lines
OK. 67553994410557440 is 61440 byte swapped in 64 bits, and 30MB is 61440 sectors, so this should be a simple byte order fix.
More strangely, the upper and lower 32-bit words are swapped. The bytes within each 32-bit word are swapped correctly.
So the calculation maybe should be something like:
__le32 *p = (__le32 *) &id[100 << 1]; u32 high32 = le32_to_cpup(p); u32 low32 = le32_to_cpup(p + 1); ssize = (((u64)high32 << 32) | (u64) low32);
Quoted 7 lines
But that doesn't make any sense, and even
ide_fix_driveid()indrivers/ide/ide-iops.cdoes ale64_to_cpu()for this value:id->lba_capacity_2 = __le64_to_cpu(id->lba_capacity_2);I wonder if this is some artifact of how AOE devices encode this field when sending it to the client.
Well, an EtherDrive blade just copies the ATA identify response data into a network packet without looking at it. The vblade, though, has to set the lba_capacity and lba_capacity_2 fields itself.
The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.
The vblade bug was the omission of swapping the bytes in each short. The fix below shows what I mean:
ata.c +18 -13
@@ -55,24 +55,29 @@ setfld(ushort *a, int idx, int len, char }static void -setlba28 (ushort *p,vlong lba) +setlba28 (ushort *ident,vlong lba) { - p +=60 ; - *p++ = lba &0xffff ; - *p = lba >>16 &0x0fffffff ; +uchar *cp; + + cp = (uchar *) &ident[60 ]; + *cp++ = lba; + *cp++ = lba >>=8 ; + *cp++ = lba >>=8 ; + *cp++ = (lba >>=8 ) &0xf ; }static void -setlba48 (ushort *p,vlong lba) +setlba48 (ushort *ident,vlong lba) { - p +=100 ; - *p++ = lba; - lba >>=16 ; - *p++ = lba; - lba >>=16 ; - *p++ = lba; - lba >>=16 ; - *p = lba; +uchar *cp; + + cp = (uchar *) &ident[100 ]; + *cp++ = lba; + *cp++ = lba >>=8 ; + *cp++ = lba >>=8 ; + *cp++ = lba >>=8 ; + *cp++ = lba >>=8 ; + *cp++ = lba >>=8 ; }void
Signature · 1 line
Ed L Cashin <[email protected]>
From: Ed L Cashin <[email protected]>
Date: Thu, 01 Sep 2005 15:13:52 -0400
Quoted 4 lines
The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.
I see, thanks for looking into this.
Quoted 4 lines · Ed L Cashin
The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.
It is in fact a x86_64 kernel, but with a mostly x86 userland. Vblade is pure x86 code.
Quoted 2 lines
The vblade bug was the omission of swapping the bytes in each short. The fix below shows what I mean:
Unfortunately it doesn't fix anything here. The client still reports the same wrong size as before. The dmesg output is identical, too.
Regards,
Jim
Quoted 7 lines · Jim MacBaine
Quoted 4 lines · Ed L Cashin
The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.
Quoted 8 lines
It is in fact a
x86_64kernel, but with a mostly x86 userland. Vblade is pure x86 code.Quoted 2 lines
The vblade bug was the omission of swapping the bytes in each short. The fix below shows what I mean:
Unfortunately it doesn't fix anything here. The client still reports the same wrong size as before. The dmesg output is identical, too.
Let's take this discussion off the lkml, because I doubt there's a problem with the aoe driver in the kernel, and I can easily follow up to the lkml with a synopsis if it turns out I'm wrong.
Jim MacBaine, I'm going to ask for more details in a separate email.
Signature · 1 line
Ed L Cashin <[email protected]>
Ed L Cashin <[email protected]> writes:
...
Quoted 3 lines
Let's take this discussion off the lkml, because I doubt there's a problem with the aoe driver in the kernel, and I can easily follow up to the lkml with a synopsis if it turns out I'm wrong.
It looks like I was probably wrong. I need to do some debugging, but the only sparc64 machine here at hand is in use.
If anybody would be up for running 2.6.13 on a sparc64 host and running tests with a patched aoe driver, please let me know. A test would look something like this, using an x86 host and a sparc64 host on the same LAN.
x86$ dd if=/dev/zero of=/tmp/0x1234567 bs=1k count=1 seek=19088742 x86$ vblade 0 1 eth0 /tmp/0x1234567
sparc64$ rmmod aoe sparc64$ cd ~/linux-2.6.13 sparc64$ patch -p1 < aoe.diff sparc64$ make && make modules_install sparc64$ modprobe aoe
I'd email you patches, and you'd email me the printk messages that show up in the logs. Such help would be much appreciated.
Signature · 1 line
Ed L Cashin <[email protected]>
Quoted 16 lines · Jim MacBaine
Hello,
Using aoe on a sparc64 system gives strange results:
sunny:/dev/etherd# echo >discover sunny:/dev/etherd# mke2fs e0.0 mke2fs 1.37 (21-Mar-2005) mke2fs: File too large while trying to determine filesystem size sunny:/dev/etherd# blockdev --getsz e0.0 -4503599627370496The log says:
Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6> etherd/e0.0: unknown partition table Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has
Quoted 6 lines
67553994410557440
sectorsThe system is an Sun Ultra 5, running 2.6.12.5/sparc64 compiled with gcc-3.4.2. e0.0 is exported on a x86 system using vblade-5, and has a size of 30 MB.
I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test le64_to_cpup by itself, it works as expected.
drivers/block/aoe/aoecmd.c +2 -1
@@ -320,7 +320,8 @@ d->flags |=DEVFL_EXT ;/* word 100: number lba48 sectors */ - ssize =le64_to_cpup ((__le64 *) &id[100 <<1 ]); + ssize = *((u64 *) &id[100 <<1 ]); + ssize =le64_to_cpu (ssize);/* set as in ide-disk.c:init_idedisk_capacity */ d->geo.cylinders = ssize;
Signature · 2 lines
-- Ed L Cashin <[email protected]>
From: Ed L Cashin <[email protected]>
Date: Fri, 16 Sep 2005 09:36:51 -0400
Quoted 3 lines
I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test
le64_to_cpupby itself, it works as expected.
This reminds me of a known UltraSPARC chip bug. There is a bug wherein if you do a 64-bit endianness swapped load instruction, it only endian swaps within each 32-bit word, not throughout the entire 64-bit word as it should.
But this is weird, because the errata description for this chip bug says that it only applies to the deprecated 32-bit mode "ldd/std" instructions, whereas the compiler emits the 64-bit "ldx/stx" instructions for sparc64 kernel builds.
So, first, let's sanity check the ___arch__swab64p() implementation:
#include <stdlib.h>
#include <stdio.h>
#define ASI_PL 0x88 /* Primary, implicit, l-endian */
typedef unsigned long __u64;
static __inline__ __u64 ___arch__swab64p(const __u64 *addr)
{
__u64 ret;
__asm__ __volatile__ ("ldxa [%1] %2, %0"
: "=r" (ret)
: "r" (addr), "i" (ASI_PL));
return ret;
}int main(void)
{
unsigned long tval = 0x123456789abcdef0;
unsigned long *p, v;
p = &tval;
v = ___arch__swab64p(p);
printf("%016lx --> %016lx\n",
tval, v);
exit(0);
}Running this on my Ultra-IIIi workstation (this chip doesn't have said errata) gives the desired result:
davem@sunset:~/src/GIT/sparc-2.6$ gcc -m64 -O2 -o x x.c davem@sunset:~/src/GIT/sparc-2.6$ ./x 123456789abcdef0 --> f0debc9a78563412 davem@sunset:~/src/GIT/sparc-2.6$
So it looks sane. Let's try this on a chip that has the errata:
? ./x123456789abcdef0 --> f0debc9a78563412
?
That's fine too, so this isn't what is causing the problems.
Wait, I think I see the problem. Is that "&id[100<<1]" pointer aligned properly on a 64-bit boundary? I bet it isn't, and the sparc64 unaligned load/store trap handler doesn't check whether one of the endian swapping load/store attributes are being used.
Looking at the assembler generated for aoecmd.s, it isn't aligned:
add %l1, 236, %g2 ... ldxa [%g2] 136, %g7
That's aligned on a 4-byte boundary, but not an 8-byte one.
So this is what the bug is.
From: Ed L Cashin <[email protected]>
Date: Fri, 16 Sep 2005 09:36:51 -0400
Quoted 3 lines
I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test
le64_to_cpupby itself, it works as expected.
This patch should fix the bug.
arch/sparc64/kernel/una_asm.S +1 -1
@@ -17,7 +17,7 @@ kernel_unaligned_trap_fault: __do_int_store: rd %asi, %o4 wr %o3, 0, %asi - ldx [%o2], %g3 + mov %o2, %g3 cmp %o1, 2 be,pn %icc, 2f cmp %o1, 4
arch/sparc64/kernel/unaligned.c +57 -7
101 lines not on this page · raw message
Quoted 3 lines · David S. Miller
This patch should fix the bug.
arch/sparc64/kernel/una_asm.S +0
-0
[..]
arch/sparc64/kernel/unaligned.c
Quoted 1 line
arch/sparc64/kernel/unaligned.c +0
-0
[..]
Was this patch meant to be applied to a fresh 2.6.13 kernel without any of Ed's patches? If so, I cannot confirm that this patch works. The aoe driver still reports a wrong size:
sunny:~# modprobe aoe aoe: aoe_init: AoE v2.6-10 initialised. etherd/e0.0: unknown partition table aoe: 0011xxxxxxxx e0.0 v4000 has 7441392446501552128 sectors
The exported file has got a size of 19088743 sectors.
Regards,
Jim
From: Jim MacBaine <[email protected]>
Date: Sat, 17 Sep 2005 12:10:17 +0200
Quoted 3 lines
Was this patch meant to be applied to a fresh 2.6.13 kernel without any of Ed's patches? If so, I cannot confirm that this patch works. The aoe driver still reports a wrong size:
Please double check that you really ran with the patch applied. I even wrote a test kernel module that verified that the bug was fixed by doing various unaligned 64-bit loads and stores, both with and without endianness swapping. It definitely failed before the patch, and definitely worked with the patch.
arch/sparc64/kernel/una_asm.S
Quoted 12 lines · David S. Miller
From: Ed L Cashin <[email protected]>
Date: Fri, 16 Sep 2005 09:36:51 -0400Quoted 3 lines
I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test
le64_to_cpupby itself, it works as expected.This patch should fix the bug.
arch/sparc64/kernel/una_asm.S +0
-0
So it's OK to use the "...._to_cpup" macros with unaligned pointers? I'm asking whether ...
1) Passing le64_to_cpup an unaligned pointer is "OK" and within the intended use of the function. I'm having trouble finding whether this is documented somewhere.
2) These new changes to the sparc64 unaligned access fault handling will make it OK to leave the aoe driver the way it is in the mainline kernel.
...
Quoted 1 line
arch/sparc64/kernel/unaligned.c +0
-0
Signature · 1 line
Ed L Cashin <[email protected]>
From: Ed L Cashin <[email protected]>
Date: Mon, 19 Sep 2005 10:24:00 -0400
Quoted 7 lines
1) Passing
le64_to_cpupan unaligned pointer is "OK" and within the intended use of the function. I'm having trouble finding whether this is documented somewhere.2) These new changes to the sparc64 unaligned access fault handling will make it OK to leave the aoe driver the way it is in the mainline kernel.
Both #1 and #2 are true.
Although it's very much discouraged to dereference unaligned pointers, especially in performance critical code (which this AOE case is not, thankfully), because performance will be really bad as the trap handler has to fix up the access on RISC platforms.
David> Although it's very much discouraged to dereference David> unaligned pointers, especially in performance critical code David> (which this AOE case is not, thankfully), because David> performance will be really bad as the trap handler has to David> fix up the access on RISC platforms.
Also, ia64 has a tendency to print an ugly message in the kernel log for unaligned accesses. Has anyone tried AoE on ia64?
It might be better to change the AoE code to use get_unaligned(), just to document what's going on. Although clearly the sparc64 patch is correct as well -- we should never silently return the wrong data.
- R.
Quoted 7 lines · David S. Miller
From: Ed L Cashin <[email protected]>
Date: Mon, 19 Sep 2005 10:24:00 -0400Quoted 4 lines
1) Passing
le64_to_cpupan unaligned pointer is "OK" and within the intended use of the function. I'm having trouble finding whether this is documented somewhere.
Quoted 5 lines
Quoted 3 lines
2) These new changes to the sparc64 unaligned access fault handling will make it OK to leave the aoe driver the way it is in the mainline kernel.
Both #1 and #2 are true.
That's interesting. I think I'll send a patch documenting #1.
Quoted 4 lines
Although it's very much discouraged to dereference unaligned pointers, especially in performance critical code (which this AOE case is not, thankfully), because performance will be really bad as the trap handler has to fix up the access on RISC platforms.
Yes, this only happens when per AoE device when the AoE device is discovered. Still, I might submit a patch that reverts the aoe driver to getting the ATA identify values byte by byte as it used to do.
Signature · 1 line
Ed L Cashin <[email protected]>