64-bit fix

This patch adds detection of broken 64-bit mode LL/SC on Cobalt units. With this patch my Qube2700 boots a 64-bit build fine. The later units have some problems with the Tulip driver.

P.

--

Index: linux/arch/mips/kernel/cpu-probe.c
===================================================================
@@ -131,6 +131,58 @@
 	check_wait();
 }
 
+#ifdef CONFIG_MIPS64
+
+/*
+ * On RM5230/5231 all accesses to XKPHYS by LL(D) are forced
+ * to be uncached, bits 61-59 of the address are ignored.
+ *
+ * Apparently fixed on RM5230A/5231A.
+ */
+static inline int check_lld(void)
+{
+	unsigned long flags, value, match, phys, *addr;
+
+	printk("Checking for lld bug... ");
+
+	/* hope the stack is in the low 512MB */
+	phys = CPHYSADDR((unsigned long) &value);
+
+	/* write value to memory */
+	value = 0xfedcba9876543210;
+	addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_UNCACHED, phys);
+	*addr = value;
+
+	/* stop spurious flushes */
+	local_irq_save(flags);
+
+	/* flip cached value */
+	value = ~value;
+
+	/* read value, supposedly from cache */
+	addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_NONCOHERENT, phys);
+	asm volatile("lld %0, %1" : "=r" (match) : "m" (*addr));
+
+	local_irq_restore(flags);
+
+	match ^= value;
+
+	switch ((long) match) {
+	case 0:
+		printk("no.\n");
+		break;
+	case -1:
+		printk("yes.\n");
+		break;
+	default:
+		printk("yikes yes! (%lx/%lx@%p)\nPlease report to <[email protected]>.", value, match, &value);
+	}
+
+	return !match;
+}
+
+#endif
+
 /*
  * Probe whether cpu has config register by trying to play with
  * alternate cache bit and see whether it matters.

@@ -347,7 +399,11 @@
 		c->cputype = CPU_NEVADA;
 		c->isa_level = MIPS_CPU_ISA_IV;
 		c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-		             MIPS_CPU_DIVEC | MIPS_CPU_LLSC;
+		             MIPS_CPU_DIVEC;
+#ifdef CONFIG_MIPS64
+		if (check_lld())
+#endif
+			c->options |= MIPS_CPU_LLSC;
 		c->tlbsize = 48;
 		break;
 	case PRID_IMP_R6000:

Index: linux/include/asm-mips/addrspace.h
===================================================================
@@ -120,7 +120,7 @@
 #define PHYS_TO_XKSEG_UNCACHED(p)	PHYS_TO_XKPHYS(K_CALG_UNCACHED,(p))
 #define PHYS_TO_XKSEG_CACHED(p)		PHYS_TO_XKPHYS(K_CALG_COH_SHAREABLE,(p))
 #define XKPHYS_TO_PHYS(p)		((p) & TO_PHYS_MASK)
-#define PHYS_TO_XKPHYS(cm,a)		(0x8000000000000000 | ((cm)<<59) | (a))
+#define PHYS_TO_XKPHYS(cm,a)		(0x8000000000000000 | ((unsigned long)(cm)<<59) | (a))
 
 #if defined (CONFIG_CPU_R4300)						\
     || defined (CONFIG_CPU_R4X00)					\

Just out of curiosity, is there any practical interest in going 64bit on Cobalt besides the fun of it? One cannot possibly squeeze more than 4 Gb of RAM into a Cobalt box right? And doesn't 64 bit mode have costs of its own (doubled i-fetch bandwidth for starters)?

<< Tout n'y est pas parfait, mais on y honore certainement les jardiniers >>

			Dominique Quatravaux <[email protected]>

Second hand Cobalt MIPS hardware is available fairly cheaply so it's being used by various Linux developers for doing development work, including 64-bit work.

No, the limit is significantly less. 64-bit kernels are advantagous if

- running N32 or N64 software is desired
- anything that takes advantage of 64-bit registers or the 32/32 fpr model
- software is using large amounts of virtual address space. Process size is limited to 2GB which is tight for some of todays codes which do their I/O by memory mapping files.
- and of the course there is the "more inches" factor ;-)

Fortunately not double and caches will further blurr the picture - but on a system with a 32-bit processor and memory bus there will be very noticable impact. We're using a bunch of tricks to keep the overhead under control.

Ralf

64-bit MIPS CPUs still have 32-bit instructions... it's the registers and addressing range which grow, not the instructions.

Program data segments tend to grow when you use 64-bit pointers (N64 does, but N32 - paradoxically still a 64-bit ABI - doesn't)

--
Dominic Sweetman
MIPS Technologies

True - but number of instructions will grow, thus the bandwidth needed to fetch them.

N32 is an ILP32 ABI so I'd count it as 32-bit.

Ralf

Ralf, is this patch appropriate? Can you please apply it or provide some feedback.

Martin Michlmayr
http://www.cyrius.com/

Here's a link to the updated patch. Works under 2.6.15
http://www.linuxfromscratch.org/patches/downloads/linux/linux-2.6.15-mips_fix-1.patch

This include the iomap.c, which is not accepted by Ralf.

Submitted By: Jim Gifford (patches at jg555 dot com)
Date: 2006-01-09
Initial Package Version: 2.6.15
Origin: Jim Gifford
Upstream Status: Not Accepted
Description: Various Fixes for MIPS architectures

diff -Naur linux-2.6.15.orig/arch/mips/lib/Makefile linux-2.6.15/arch/mips/lib/Makefile
@@ -5,4 +5,6 @@

lib-y += csum_partial_copy.o memcpy.o promlib.o strlen_user.o strncpy_user.o \
strnlen_user.o uncached.o

+obj-y += iomap.o
+
EXTRA_AFLAGS := $(CFLAGS)

diff -Naur linux-2.6.15.orig/include/asm-mips/addrspace.h linux-2.6.15/include/asm-mips/addrspace.h
@@ -124,7 +124,7 @@

#define PHYS_TO_XKSEG_CACHED(p) PHYS_TO_XKPHYS(K_CALG_COH_SHAREABLE,(p))
#define XKPHYS_TO_PHYS(p) ((p) & TO_PHYS_MASK)
#define PHYS_TO_XKPHYS(cm,a) (_LLCONST_(0x8000000000000000) | \
- ((cm)<<59) | (a))
+ ((unsigned long)(cm)<<59) | (a))

#if defined (CONFIG_CPU_R4300)                        \
    || defined (CONFIG_CPU_R4X00)                    \

diff -Naur linux-2.6.15.orig/include/asm-mips/cobalt/cpu-feature-overrides.h linux-2.6.15/include/asm-mips/cobalt/cpu-feature-overrides.h
@@ -0,0 +1,17 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2006 Ralf Baechle
+ */
+#ifndef __ASM_COBALT_CPU_FEATURE_OVERRIDES_H
+#define __ASM_COBALT_CPU_FEATURE_OVERRIDES_H
+
+#ifdef CONFIG_64BIT
+#define cpu_has_llsc            0
+#else
+#define cpu_has_llsc            1
+#endif
+
+#endif /* __ASM_COBALT_CPU_FEATURE_OVERRIDES_H */

diff -Naur linux-2.6.15.orig/include/asm-mips/io.h linux-2.6.15/include/asm-mips/io.h
@@ -535,6 +535,62 @@

}

/*
+ * Memory Mapped I/O
+ */
+#define ioread8(addr) readb(addr)
+#define ioread16(addr) readw(addr)
+#define ioread32(addr) readl(addr)
+
+#define iowrite8(b,addr) writeb(b,addr)
+#define iowrite16(w,addr) writew(w,addr)
+#define iowrite32(l,addr) writel(l,addr)
+
+#define ioread8_rep(a,b,c) readsb(a,b,c)
+#define ioread16_rep(a,b,c) readsw(a,b,c)
+#define ioread32_rep(a,b,c) readsl(a,b,c)
+
+#define iowrite8_rep(a,b,c) writesb(a,b,c)
+#define iowrite16_rep(a,b,c) writesw(a,b,c)
+#define iowrite32_rep(a,b,c) writesl(a,b,c)
+
+/* Create a virtual mapping cookie for an IO port range */
+extern void __iomem *ioport_map(unsigned long port, unsigned int nr); +extern void ioport_unmap(void __iomem *);
+
+/* Create a virtual mapping cookie for a PCI BAR (memory or IO) */ +struct pci_dev;
+extern void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max); +extern void pci_iounmap(struct pci_dev *dev, void __iomem *);
+
+/*
+ * Memory Mapped I/O
+ */
+#define ioread8(addr) readb(addr)
+#define ioread16(addr) readw(addr)
+#define ioread32(addr) readl(addr)
+
+#define iowrite8(b,addr) writeb(b,addr)
+#define iowrite16(w,addr) writew(w,addr)
+#define iowrite32(l,addr) writel(l,addr)
+
+#define ioread8_rep(a,b,c) readsb(a,b,c)
+#define ioread16_rep(a,b,c) readsw(a,b,c)
+#define ioread32_rep(a,b,c) readsl(a,b,c)
+
+#define iowrite8_rep(a,b,c) writesb(a,b,c)
+#define iowrite16_rep(a,b,c) writesw(a,b,c)
+#define iowrite32_rep(a,b,c) writesl(a,b,c)
+
+/* Create a virtual mapping cookie for an IO port range */
+extern void __iomem *ioport_map(unsigned long port, unsigned int nr); +extern void ioport_unmap(void __iomem *);
+
+/* Create a virtual mapping cookie for a PCI BAR (memory or IO) */ +struct pci_dev;
+extern void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max); +extern void pci_iounmap(struct pci_dev *dev, void __iomem *);
+
+/*
* ISA space is 'always mapped' on currently supported MIPS systems, no
need
* to explicitly ioremap() it. The fact that the ISA IO space is mapped
* to PAGE_OFFSET is pure coincidence - it does not mean ISA values

----
Jim Gifford
[email protected]

I suppose that's just an omission. Here's Yuasa's patch again (with a minor change so it applies to 2.6.15).

iomap implementation for Linux/MIPS

diff -urN linux-mips/arch/mips/lib/Makefile new/arch/mips/lib/Makefile
@@ -2,7 +2,7 @@
 # Makefile for MIPS-specific library files..
 #
 
-lib-y	+= csum_partial_copy.o memcpy.o promlib.o strlen_user.o strncpy_user.o \
-	   strnlen_user.o uncached.o
+lib-y	+= csum_partial_copy.o iomap.o memcpy.o promlib.o strlen_user.o \
+	   strncpy_user.o strnlen_user.o uncached.o
 
 EXTRA_AFLAGS := $(CFLAGS)

diff -urN linux-mips/include/asm-mips/io.h new/include/asm-mips/io.h
@@ -535,6 +535,34 @@
 }
 
 /*
+ * Memory Mapped I/O
+ */
+#define ioread8(addr)		readb(addr)
+#define ioread16(addr)		readw(addr)
+#define ioread32(addr)		readl(addr)
+
+#define iowrite8(b,addr)	writeb(b,addr)
+#define iowrite16(w,addr)	writew(w,addr)
+#define iowrite32(l,addr)	writel(l,addr)
+
+#define ioread8_rep(a,b,c)	readsb(a,b,c)
+#define ioread16_rep(a,b,c)	readsw(a,b,c)
+#define ioread32_rep(a,b,c)	readsl(a,b,c)
+
+#define iowrite8_rep(a,b,c)	writesb(a,b,c)
+#define iowrite16_rep(a,b,c)	writesw(a,b,c)
+#define iowrite32_rep(a,b,c)	writesl(a,b,c)
+
+/* Create a virtual mapping cookie for an IO port range */
+extern void __iomem *ioport_map(unsigned long port, unsigned int nr);
+extern void ioport_unmap(void __iomem *);
+
+/* Create a virtual mapping cookie for a PCI BAR (memory or IO) */
+struct pci_dev;
+extern void __iomem *pci_iomap(struct pci_dev *dev, int bar, unsigned long max);
+extern void pci_iounmap(struct pci_dev *dev, void __iomem *);
+
+/*
  * ISA space is 'always mapped' on currently supported MIPS systems, no need
  * to explicitly ioremap() it. The fact that the ISA IO space is mapped
  * to PAGE_OFFSET is pure coincidence - it does not mean ISA values

Martin Michlmayr
http://www.cyrius.com/

Yes - and the reasons are archived on this list. Reposting the patch leaves me entirely unimpressed.

Ralf

ralf> Yes - and the reasons are archived on this list. Reposting the ralf> patch leaves me entirely unimpressed.

Ralf, is this the reason?

It seems a bit obscured for me --- and perhaps for some other people. So I asked:

But got no response at that time. So I ask again. Could you tell us how the iomap patch broken verbosely, please?

---
Atsushi Nemoto

How can we get this resolved, this issue has been open a long time. Can we all work together to get a working solution in place that everyone will accept?

We all need to understand the concerns with the current method. The only issue I see from Ralf is the following:

Broken on multiple PCI busses.

Now the way I understand the issue is the current iomap.c only
handles a single bus, Ralf's point is that if there are multiple busses this patch may not work properly. Is that a correct statement Ralf.

So can't we have one iomap.c for single pci bus systems and one for
multiple pci bus systems? Just a thought.

----
Jim Gifford
[email protected]

jim> We all need to understand the concerns with the current jim> method. The only issue I see from Ralf is the following:

jim> Broken on multiple PCI busses.

Yes, it is an only reason I have ever seen.

jim> Now the way I understand the issue is the current iomap.c
jim> only handles a single bus, Ralf's point is that if there are jim> multiple busses this patch may not work properly. Is that a jim> correct statement Ralf.

I think the current iomap can handle multiple busses.

The traditional way (ioremap + read[bwl] for memory space and in[bwl] for IO space) can be used on multiple PCI busses if pci resources were properly configured. (Though IO space address might be a bit strange value due to single global mips_io_port_base, it works anyway.) The current iomap basically do same jobs.

---
Atsushi Nemoto

As per our conversation Ralf, here is the patch without the iomap stuff in it.

----
Jim Gifford
[email protected]

Submitted By: Jim Gifford (patches at jg555 dot com)
Date: 2006-01-09
Initial Package Version: 2.6.15
Origin: Jim Gifford
Upstream Status: Not Accepted
Description: Various Fixes for MIPS architectures

diff -Naur linux-2.6.15.orig/include/asm-mips/addrspace.h linux-2.6.15/include/asm-mips/addrspace.h
@@ -124,7 +124,7 @@
 #define PHYS_TO_XKSEG_CACHED(p)		PHYS_TO_XKPHYS(K_CALG_COH_SHAREABLE,(p))
 #define XKPHYS_TO_PHYS(p)		((p) & TO_PHYS_MASK)
 #define PHYS_TO_XKPHYS(cm,a)		(_LLCONST_(0x8000000000000000) | \
-					 ((cm)<<59) | (a))
+					 ((unsigned long)(cm)<<59) | (a))
 
 #if defined (CONFIG_CPU_R4300)						\
     || defined (CONFIG_CPU_R4X00)					\

diff -Naur linux-2.6.15.orig/include/asm-mips/cobalt/cpu-feature-overrides.h linux-2.6.15/include/asm-mips/cobalt/cpu-feature-overrides.h
@@ -0,0 +1,17 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2006 Ralf Baechle
+ */
+#ifndef __ASM_COBALT_CPU_FEATURE_OVERRIDES_H
+#define __ASM_COBALT_CPU_FEATURE_OVERRIDES_H
+
+#ifdef CONFIG_64BIT
+#define cpu_has_llsc            0
+#else
+#define cpu_has_llsc            1
+#endif
+
+#endif /* __ASM_COBALT_CPU_FEATURE_OVERRIDES_H */

Runtime testing for that bug is fairly expensive as it adds a branch to every instance of every type of atomic operation. So we really want cpu_has_llsc to be a constant so the compiler can optimize that.

So I suggest something like this in cpu-feature-overrides.h:

[...]
/*
* R5000 has an interesting "restriction": ll(d)/sc(d)
* instructions to XKPHYS region simply do uncached bus
* requests. This breaks all the atomic bitops functions.
* so, for 64bit IP32 kernel we just don't use ll/sc.
* This does not affect luserland.
*/
#ifdef CONFIG_64BIT
#define cpu_has_llsc 0
#else
#define cpu_has_llsc 1
#endif
[...]

The probe would upset the IP27 cache coherency logic and crash them btw.

Ralf