Skip to contentSkip to conversation

aoe fails on sparc64

DISCUSSION Jim MacBaine 4 participants 17 messages Aug 31, 2005 – Sep 19, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

Using aoe on a sparc64 system gives strange results:

sunny:/dev/etherd# echo >discover
sunny:/dev/etherd# mke2fs e0.0
mke2fs 1.37 (21-Mar-2005)
mke2fs: File too large while trying to determine filesystem size
sunny:/dev/etherd# blockdev --getsz e0.0
-4503599627370496

The log says:

Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6>
etherd/e0.0: unknown partition table
Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has

67553994410557440
sectors

The system is an Sun Ultra 5, running 2.6.12.5/sparc64 compiled with gcc-3.4.2. e0.0 is exported on a x86 system using vblade-5, and has a size of 30 MB.

Regards,
Jim

#1 lore raw
Quoted 4 lines · Jim MacBaine

Hello,

Using aoe on a sparc64 system gives strange results:

...

Quoted 7 lines

The log says:

Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6>
etherd/e0.0: unknown partition table
Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has

67553994410557440
sectors

OK. 67553994410557440 is 61440 byte swapped in 64 bits, and 30MB is 61440 sectors, so this should be a simple byte order fix.

Quoted 3 lines

The system is an Sun Ultra 5, running 2.6.12.5/sparc64 compiled with gcc-3.4.2. e0.0 is exported on a x86 system using vblade-5, and has a size of 30 MB.

Thanks for the report.

Signature · 1 line
  Ed L Cashin <[email protected]>
#2 lore raw

From: Ed L Cashin <[email protected]>
Date: Wed, 31 Aug 2005 11:50:55 -0400

Quoted 10 lines
Quoted 5 lines · Jim MacBaine
Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6>
etherd/e0.0: unknown partition table
Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has

67553994410557440
sectors

OK. 67553994410557440 is 61440 byte swapped in 64 bits, and 30MB is 61440 sectors, so this should be a simple byte order fix.

More strangely, the upper and lower 32-bit words are swapped. The bytes within each 32-bit word are swapped correctly.

So the calculation maybe should be something like:

	__le32 *p = (__le32 *) &id[100 << 1];
	u32 high32 = le32_to_cpup(p);
	u32 low32 = le32_to_cpup(p + 1);

	ssize = (((u64)high32 << 32) | (u64) low32);

But that doesn't make any sense, and even ide_fix_driveid() in drivers/ide/ide-iops.c does a le64_to_cpu() for this value:

	id->lba_capacity_2 = __le64_to_cpu(id->lba_capacity_2);

I wonder if this is some artifact of how AOE devices encode this field when sending it to the client.

#3 lore raw
Quoted 1 line · David S. Miller

From: Ed L Cashin <[email protected]>

...

Quoted 14 lines
Quoted 2 lines

OK. 67553994410557440 is 61440 byte swapped in 64 bits, and 30MB is 61440 sectors, so this should be a simple byte order fix.

More strangely, the upper and lower 32-bit words are swapped. The bytes within each 32-bit word are swapped correctly.

So the calculation maybe should be something like:

	__le32 *p = (__le32 *) &id[100 << 1];
	u32 high32 = le32_to_cpup(p);
	u32 low32 = le32_to_cpup(p + 1);

	ssize = (((u64)high32 << 32) | (u64) low32);
Quoted 7 lines

But that doesn't make any sense, and even ide_fix_driveid() in drivers/ide/ide-iops.c does a le64_to_cpu() for this value:

	id->lba_capacity_2 = __le64_to_cpu(id->lba_capacity_2);

I wonder if this is some artifact of how AOE devices encode this field when sending it to the client.

Well, an EtherDrive blade just copies the ATA identify response data into a network packet without looking at it. The vblade, though, has to set the lba_capacity and lba_capacity_2 fields itself.

The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.

The vblade bug was the omission of swapping the bytes in each short. The fix below shows what I mean:

ata.c +18 -13
diff -urNp a-exp/ata.c b-exp/ata.c
@@ -55,24 +55,29 @@ setfld(ushort *a, int idx, int len, char
 }
 
 static void
-setlba28(ushort *p, vlong lba)
+setlba28(ushort *ident, vlong lba)
 {
-	p += 60;
-	*p++ = lba & 0xffff;
-	*p = lba >> 16 & 0x0fffffff;
+	uchar *cp;
+
+	cp = (uchar *) &ident[60];
+	*cp++ = lba;
+	*cp++ = lba >>= 8;
+	*cp++ = lba >>= 8;
+	*cp++ = (lba >>= 8) & 0xf;
 }
 
 static void
-setlba48(ushort *p, vlong lba)
+setlba48(ushort *ident, vlong lba)
 {
-	p += 100;
-	*p++ = lba;
-	lba >>= 16;
-	*p++ = lba;
-	lba >>= 16;
-	*p++ = lba;
-	lba >>= 16;
-	*p = lba;
+	uchar *cp;
+
+	cp = (uchar *) &ident[100];
+	*cp++ = lba;
+	*cp++ = lba >>= 8;
+	*cp++ = lba >>= 8;
+	*cp++ = lba >>= 8;
+	*cp++ = lba >>= 8;
+	*cp++ = lba >>= 8;
 }
 		
 void
Signature · 1 line
  Ed L Cashin <[email protected]>
#4 lore raw

From: Ed L Cashin <[email protected]>
Date: Thu, 01 Sep 2005 15:13:52 -0400

Quoted 4 lines

The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.

I see, thanks for looking into this.

#5 lore raw
In reply to Ed L Cashin
Quoted 4 lines · Ed L Cashin

The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.

It is in fact a x86_64 kernel, but with a mostly x86 userland. Vblade is pure x86 code.

Quoted 2 lines

The vblade bug was the omission of swapping the bytes in each short. The fix below shows what I mean:

Unfortunately it doesn't fix anything here. The client still reports the same wrong size as before. The dmesg output is identical, too.

Regards,
Jim

#6 lore raw
Quoted 7 lines · Jim MacBaine
Quoted 4 lines · Ed L Cashin

The aoe driver looks OK, but it turns out there's a byte swapping bug in the vblade that could be related if he's running the vblade on a big endian host (even though he said it was an x86 host), but I haven't heard back from the original poster yet.

Quoted 8 lines

It is in fact a x86_64 kernel, but with a mostly x86 userland. Vblade is pure x86 code.

Quoted 2 lines

The vblade bug was the omission of swapping the bytes in each short. The fix below shows what I mean:

Unfortunately it doesn't fix anything here. The client still reports the same wrong size as before. The dmesg output is identical, too.

Let's take this discussion off the lkml, because I doubt there's a problem with the aoe driver in the kernel, and I can easily follow up to the lkml with a synopsis if it turns out I'm wrong.

Jim MacBaine, I'm going to ask for more details in a separate email.

Signature · 1 line
  Ed L Cashin <[email protected]>
#7 lore raw

Ed L Cashin <[email protected]> writes:

...

Quoted 3 lines

Let's take this discussion off the lkml, because I doubt there's a problem with the aoe driver in the kernel, and I can easily follow up to the lkml with a synopsis if it turns out I'm wrong.

It looks like I was probably wrong. I need to do some debugging, but the only sparc64 machine here at hand is in use.

If anybody would be up for running 2.6.13 on a sparc64 host and running tests with a patched aoe driver, please let me know. A test would look something like this, using an x86 host and a sparc64 host on the same LAN.

x86$ dd if=/dev/zero of=/tmp/0x1234567 bs=1k count=1 seek=19088742 x86$ vblade 0 1 eth0 /tmp/0x1234567

  sparc64$ rmmod aoe
  sparc64$ cd ~/linux-2.6.13
  sparc64$ patch -p1 < aoe.diff
  sparc64$ make && make modules_install
  sparc64$ modprobe aoe

I'd email you patches, and you'd email me the printk messages that show up in the logs. Such help would be much appreciated.

Signature · 1 line
  Ed L Cashin <[email protected]>
#8 lore raw
In reply to Jim MacBaine
Quoted 16 lines · Jim MacBaine

Hello,

Using aoe on a sparc64 system gives strange results:

sunny:/dev/etherd# echo >discover
sunny:/dev/etherd# mke2fs e0.0
mke2fs 1.37 (21-Mar-2005)
mke2fs: File too large while trying to determine filesystem size
sunny:/dev/etherd# blockdev --getsz e0.0
-4503599627370496

The log says:

Aug 31 15:18:49 sunny kernel: devfs_mk_dir: invalid argument.<6>
etherd/e0.0: unknown partition table
Aug 31 15:18:49 sunny kernel: aoe: 0011d8xxxxxx e0.0 v4000 has
Quoted 6 lines

67553994410557440
sectors

The system is an Sun Ultra 5, running 2.6.12.5/sparc64 compiled with gcc-3.4.2. e0.0 is exported on a x86 system using vblade-5, and has a size of 30 MB.

I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test le64_to_cpup by itself, it works as expected.

drivers/block/aoe/aoecmd.c +2 -1
Index: linux-2.6.13/drivers/block/aoe/aoecmd.c
===================================================================
@@ -320,7 +320,8 @@
 		d->flags |= DEVFL_EXT;
 
 		/* word 100: number lba48 sectors */
-		ssize = le64_to_cpup((__le64 *) &id[100<<1]);
+		ssize = *((u64 *) &id[100<<1]);
+		ssize = le64_to_cpu(ssize);
 
 		/* set as in ide-disk.c:init_idedisk_capacity */
 		d->geo.cylinders = ssize;
Signature · 2 lines
-- 
  Ed L Cashin <[email protected]>
#9 lore raw

From: Ed L Cashin <[email protected]>
Date: Fri, 16 Sep 2005 09:36:51 -0400

Quoted 3 lines

I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test le64_to_cpup by itself, it works as expected.

This reminds me of a known UltraSPARC chip bug. There is a bug wherein if you do a 64-bit endianness swapped load instruction, it only endian swaps within each 32-bit word, not throughout the entire 64-bit word as it should.

But this is weird, because the errata description for this chip bug says that it only applies to the deprecated 32-bit mode "ldd/std" instructions, whereas the compiler emits the 64-bit "ldx/stx" instructions for sparc64 kernel builds.

So, first, let's sanity check the ___arch__swab64p() implementation:

#include <stdlib.h>
#include <stdio.h>

#define ASI_PL 0x88 /* Primary, implicit, l-endian */

typedef unsigned long __u64;

static __inline__ __u64 ___arch__swab64p(const __u64 *addr)

{
	__u64 ret;

	__asm__ __volatile__ ("ldxa [%1] %2, %0"
			      : "=r" (ret)
			      : "r" (addr), "i" (ASI_PL));
	return ret;
}

int main(void)

{
	unsigned long tval = 0x123456789abcdef0;
	unsigned long *p, v;

	p = &tval;
	v = ___arch__swab64p(p);
	printf("%016lx --> %016lx\n",
	       tval, v);
	exit(0);
}

Running this on my Ultra-IIIi workstation (this chip doesn't have said errata) gives the desired result:

davem@sunset:~/src/GIT/sparc-2.6$ gcc -m64 -O2 -o x x.c
davem@sunset:~/src/GIT/sparc-2.6$ ./x
123456789abcdef0 --> f0debc9a78563412
davem@sunset:~/src/GIT/sparc-2.6$

So it looks sane. Let's try this on a chip that has the errata:

? ./x
123456789abcdef0 --> f0debc9a78563412
?

That's fine too, so this isn't what is causing the problems.

Wait, I think I see the problem. Is that "&id[100<<1]" pointer aligned properly on a 64-bit boundary? I bet it isn't, and the sparc64 unaligned load/store trap handler doesn't check whether one of the endian swapping load/store attributes are being used.

Looking at the assembler generated for aoecmd.s, it isn't aligned:

	add	%l1, 236, %g2
 ...
	ldxa [%g2] 136, %g7

That's aligned on a 4-byte boundary, but not an 8-byte one.

So this is what the bug is.

#10 lore raw
In reply to Ed L Cashin

From: Ed L Cashin <[email protected]>
Date: Fri, 16 Sep 2005 09:36:51 -0400

Quoted 3 lines

I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test le64_to_cpup by itself, it works as expected.

This patch should fix the bug.

arch/sparc64/kernel/una_asm.S +1 -1
@@ -17,7 +17,7 @@ kernel_unaligned_trap_fault:
 __do_int_store:
 	rd	%asi, %o4
 	wr	%o3, 0, %asi
-	ldx	[%o2], %g3
+	mov	%o2, %g3
 	cmp	%o1, 2
 	be,pn	%icc, 2f
 	 cmp	%o1, 4
arch/sparc64/kernel/unaligned.c +57 -7

101 lines not on this page · raw message

#11 lore raw
Quoted 3 lines · David S. Miller

This patch should fix the bug.

arch/sparc64/kernel/una_asm.S +0 -0

[..]

arch/sparc64/kernel/unaligned.c

Quoted 1 line
arch/sparc64/kernel/unaligned.c +0 -0

[..]

Was this patch meant to be applied to a fresh 2.6.13 kernel without any of Ed's patches? If so, I cannot confirm that this patch works. The aoe driver still reports a wrong size:

sunny:~# modprobe aoe
aoe: aoe_init: AoE v2.6-10 initialised.
 etherd/e0.0: unknown partition table
aoe: 0011xxxxxxxx e0.0 v4000 has 7441392446501552128 sectors

The exported file has got a size of 19088743 sectors.

Regards,
Jim

#12 lore raw

From: Jim MacBaine <[email protected]>
Date: Sat, 17 Sep 2005 12:10:17 +0200

Quoted 3 lines

Was this patch meant to be applied to a fresh 2.6.13 kernel without any of Ed's patches? If so, I cannot confirm that this patch works. The aoe driver still reports a wrong size:

Please double check that you really ran with the patch applied. I even wrote a test kernel module that verified that the bug was fixed by doing various unaligned 64-bit loads and stores, both with and without endianness swapping. It definitely failed before the patch, and definitely worked with the patch.

#13 lore raw
In reply to David S. Miller

arch/sparc64/kernel/una_asm.S

Quoted 12 lines · David S. Miller

From: Ed L Cashin <[email protected]>
Date: Fri, 16 Sep 2005 09:36:51 -0400

Quoted 3 lines

I've been working with Jim MacBaine, and he reports that the patch below gets rid of the problem. I don't know why. When I test le64_to_cpup by itself, it works as expected.

This patch should fix the bug.

arch/sparc64/kernel/una_asm.S +0 -0

So it's OK to use the "...._to_cpup" macros with unaligned pointers? I'm asking whether ...

1) Passing le64_to_cpup an unaligned pointer is "OK" and within the intended use of the function. I'm having trouble finding whether this is documented somewhere.

2) These new changes to the sparc64 unaligned access fault handling will make it OK to leave the aoe driver the way it is in the mainline kernel.

...

Quoted 1 line
arch/sparc64/kernel/unaligned.c +0 -0
Signature · 1 line
  Ed L Cashin <[email protected]>
#14 lore raw

From: Ed L Cashin <[email protected]>
Date: Mon, 19 Sep 2005 10:24:00 -0400

Quoted 7 lines

1) Passing le64_to_cpup an unaligned pointer is "OK" and within the intended use of the function. I'm having trouble finding whether this is documented somewhere.

2) These new changes to the sparc64 unaligned access fault handling will make it OK to leave the aoe driver the way it is in the mainline kernel.

Both #1 and #2 are true.

Although it's very much discouraged to dereference unaligned pointers, especially in performance critical code (which this AOE case is not, thankfully), because performance will be really bad as the trap handler has to fix up the access on RISC platforms.

#15 lore raw

David> Although it's very much discouraged to dereference David> unaligned pointers, especially in performance critical code David> (which this AOE case is not, thankfully), because David> performance will be really bad as the trap handler has to David> fix up the access on RISC platforms.

Also, ia64 has a tendency to print an ugly message in the kernel log for unaligned accesses. Has anyone tried AoE on ia64?

It might be better to change the AoE code to use get_unaligned(), just to document what's going on. Although clearly the sparc64 patch is correct as well -- we should never silently return the wrong data.

- R.

#16 lore raw
In reply to David S. Miller
Quoted 7 lines · David S. Miller

From: Ed L Cashin <[email protected]>
Date: Mon, 19 Sep 2005 10:24:00 -0400

Quoted 4 lines

1) Passing le64_to_cpup an unaligned pointer is "OK" and within the intended use of the function. I'm having trouble finding whether this is documented somewhere.

Quoted 5 lines
Quoted 3 lines

2) These new changes to the sparc64 unaligned access fault handling will make it OK to leave the aoe driver the way it is in the mainline kernel.

Both #1 and #2 are true.

That's interesting. I think I'll send a patch documenting #1.

Quoted 4 lines

Although it's very much discouraged to dereference unaligned pointers, especially in performance critical code (which this AOE case is not, thankfully), because performance will be really bad as the trap handler has to fix up the access on RISC platforms.

Yes, this only happens when per AoE device when the AoE device is discovered. Still, I might submit a patch that reverts the aoe driver to getting the ATA identify values byte by byte as it used to do.

Signature · 1 line
  Ed L Cashin <[email protected]>