MIPS - 64bit woes

I've been working on getting the RaQ2 to work with the current 2.6.14 kernel, but no luck at all. The last success I had was with 2.6.12.x series.

I'm looking for ideas, patches or whatever to get this working again. I know it has to do with the kernel using 32bit addressing instead of 64 bit, but have no clue where to start.

Here is what I get when I attempt to boot it.

nfs: mounted "/nfsroot"
nfs: lookup "boot"
nfs: lookup "vmlinux-2.6.14.gz"
nfs: mode <0100755>
1349KB loaded (899KB/sec)
001512e8 1381096t
nfs: unmounted "/nfsroot"

elf64: 00080000 - 003b901f (ffffffff.80357000) (ffffffff.80000000)
elf64: ffffffff.80080000 (80080000) 3170438t + 208794t
net: interface down

Thanx for all your assistance.

----
Jim Gifford
[email protected]

Hmm, I'm unable to boot a 64bit kernel on either IP27 or IP30 here. Appears to die very early in the kernel code. It looks like your kernel also dies before doing anything meaningful, so I wonder if this is some common thing.

--Kumba

Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond

I've talked to a few others, who are having similar issues also Kumba, I made a diff of 2.6.12 and 2.6.14, trying to figure out what's causing this. Looks like some major rewrites have occured in some areas.

If anyone would like to view the diff I have it on my website, http://ftp.jg555.com/mips_diff.txt

----
Jim Gifford
[email protected]

I have a 2.6.14-rc1 kernel that works great, and when I tried 2.6.14-rc2, it would have up on boot. In my case, I'm using NFS root, and the networking would just go away when the system got busy. This would cause the nfsroot to hang, and then any process that touched a file would hang. It sort of felt like interrupts stopped working, but I was not able to verify that. It seems that a serial console continued to work, so it doesn't seem like all interrupts stopped working.

I have backported nearly all of the relevent arch/mips files to my 2.6.14-rc1 tree, and everything is still happy, so I'm inclined to think it might not be something inside arch/mips.

I've stared at the diffs between rc1 and rc2 until I'm corss-eyed, but nothing has jumped out at me as being obviously broken.

Stuart

Stuart R. Anderson                               [email protected]
Network & Software Engineering                   http://www.netsweng.com/
1024D/37A79149:                                  0791 D3B8 9A4C 2CDC A31F
                                                 BD03 0A62 E534 37A7 9149

It must be platform-specific -- I haven't checked 2.6.14, but 64-bit
2.6.13 is good enough to boot into multi-user with the SWARM.

Maciej

2.6.13.4 works on all systems I can test (O2, Octane, Origin*, Indy, I2 Impact), but for me, it appears 2.6.14 dies very early in kernel initialization on both Octane and Origin. I have yet to try another, like O2, but I'm going to do that shortly in the hopes O2 may initialize an output device and give me an idea of what's going on.

* 2.6.13.4 Works on Origin, but can't allocate PCI resources for the scsi properly. 2.6.12.5 is what works best for this system.

--Kumba

Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond

Here is what I did to track down the errors.

I created a diff between the last working kernel 2.6.12 and the current kernel. Started with cpu-probe.c, that to me seemed the logical choice.

After patching the rest of files needed to support the patch in cpu-probe.c, I was finally able to produce a kernel under 2.6.12 with the same problem.

The files that I ported from 2.6.14 to 2.6.12 are the following cpu-probe.c
cpu.h
mipsregs.h
cache.c
cpu-features.h

Here is a link to the patches I used http://ftp.jg555.com/errors

Looking at mipsregs.h, something doesn't look right for a 64 bit system. But I'm not the expert.

Here are my findings, I hope someone out there has an idea.

----
Jim Gifford
[email protected]

Stanislaw pretty much figured it out last night.

See the if statement in the 'void __init cpu_cache_init(void)' function in arch/mips/mm/cache.c. IP30, IP27, and cobalt didn't define the appropriate cache type in cpu-features.h, thus failed that check and panicked. I believe it's fixed in the newest IP30 patches, of which I have yet to take a look at.

--Kumba

Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands do them because they must, while the eyes of the great are elsewhere."  --Elrond

Does look like it. I just checked the latest

----
Jim Gifford
[email protected]

I've isolated the problem to 2.6.13-rc3. What I've done is built every kernel from 2.6.13-rc1 till it failed.

Also, Ralf, when I tried using git to pull those out of the git repo, they were missing files, had to use the cvs, not sure if you can verify it. I tried 2.6.13-rc1 and 2.6.13-rc2.

----
Jim Gifford
[email protected]

Ok,
Isolated the problem down to one file, will see if this patch will
fix the issue with 2.6.14, the patch is the a diff of a file from 2.6.13-rc2 and 2.6.13-rc3 which introduced the issue. Here is a link to the patch http://ftp.jg555.com/cobalt/culprit

Will test 2.6.14 with this patch a report back.

diff -Naur linux-mips-2.6.13-rc3/arch/mips/kernel/cpu-probe.c testbed/arch/mips/kernel/cpu-probe.c
@@ -71,27 +71,11 @@
        : : "r" (au1k_wait));

}

-static int __initdata nowait = 0;
-
-int __init wait_disable(char *s)
-{
- nowait = 1;
-
- return 1;
-}
-
-__setup("nowait", wait_disable);
-
static inline void check_wait(void)

{
    struct cpuinfo_mips *c = &current_cpu_data;

    printk("Checking for 'wait' instruction... ");

- if (nowait) {
- printk (" disabled.\n");
- return;
- }
-

    switch (c->cputype) {
    case CPU_R3081:
    case CPU_R3081E:
@@ -121,7 +105,6 @@
    case CPU_24K:
    case CPU_25KF:
    case CPU_34K:
-     case CPU_PR4450:
        cpu_wait = r4k_wait;
        printk(" available.\n");
        break;
@@ -147,6 +130,58 @@
    check_wait();
}

+#ifdef CONFIG_64BIT
+
+/*
+ * On RM5230/5231 all accesses to XKPHYS by LL(D) are forced
+ * to be uncached, bits 61-59 of the address are ignored.
+ *
+ * Apparently fixed on RM5230A/5231A.
+ */
+static inline int check_lld(void)
+{
+ unsigned long flags, value, match, phys, *addr;
+
+ printk("Checking for lld bug... ");
+
+ /* hope the stack is in the low 512MB */
+ phys = CPHYSADDR((unsigned long) &value);
+

+    /* write value to memory */
+    value = 0xfedcba9876543210;
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_UNCACHED, phys);
+    *addr = value;

+
+ /* stop spurious flushes */
+ local_irq_save(flags);
+
+ /* flip cached value */
+ value = ~value;
+

+    /* read value, supposedly from cache */
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_NONCOHERENT, phys);
+    asm volatile("lld %0, %1" : "=r" (match) : "m" (*addr));

+
+ local_irq_restore(flags);
+
+ match ^= value;
+
+ switch ((long) match) {
+ case 0:
+ printk("no.\n");
+ break;
+ case -1:
+ printk("yes.\n");
+ break;
+ default:
+ printk("yikes yes! (%lx/%lx@%p)\nPlease report to <[email protected]>.", value, match, &value);
+ }
+
+ return !match;
+}
+
+#endif
+
/*
* Probe whether cpu has config register by trying to play with
* alternate cache bit and see whether it matters.
@@ -283,8 +318,7 @@

    case PRID_IMP_R4600:
        c->cputype = CPU_R4600;
        c->isa_level = MIPS_CPU_ISA_III;
-        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-                 MIPS_CPU_LLSC;
+        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_LLSC;
        c->tlbsize = 48;
        break;
    #if 0
@@ -364,7 +398,11 @@
        c->cputype = CPU_NEVADA;
        c->isa_level = MIPS_CPU_ISA_IV;
        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |

- MIPS_CPU_DIVEC | MIPS_CPU_LLSC;
+ MIPS_CPU_DIVEC;
+#ifdef CONFIG_64BIT
+ if (check_lld())
+#endif

+            c->options |= MIPS_CPU_LLSC;
        c->tlbsize = 48;
        break;
    case PRID_IMP_R6000:
@@ -509,12 +547,6 @@
        c->ases |= MIPS_ASE_SMARTMIPS;
    if (config3 & MIPS_CONF3_DSP)
        c->ases |= MIPS_ASE_DSP;

- if (config3 & MIPS_CONF3_VINT)
- c->ases |= MIPS_CPU_VINT;
- if (config3 & MIPS_CONF3_VEIC)
- c->ases |= MIPS_CPU_VEIC;
- if (config3 & MIPS_CONF3_MT)
- c->ases |= MIPS_CPU_MIPSMT;

    return config3 & MIPS_CONF_M;
}
@@ -632,21 +664,6 @@
    }
}

-static inline void cpu_probe_philips(struct cpuinfo_mips *c) -{

-    decode_configs(c);
-    switch (c->processor_id & 0xff00) {
-    case PRID_IMP_PR4450:
-        c->cputype = CPU_PR4450;
-        c->isa_level = MIPS_CPU_ISA_M32;
-        break;

- default:
- panic("Unknown Philips Core!"); /* REVISIT: die? */
- break;
- }
-}
-
-
__init void cpu_probe(void)

{
    struct cpuinfo_mips *c = &current_cpu_data;
@@ -672,9 +689,6 @@
    case PRID_COMP_SANDCRAFT:
        cpu_probe_sandcraft(c);
        break;
-     case PRID_COMP_PHILIPS:
-        cpu_probe_philips(c);
-         break;
    default:
        c->cputype = CPU_UNKNOWN;
    }

----
Jim Gifford
[email protected]

Got a fix for 2.6.14, http://ftp.jg555.com/cobalt/fix-2.6.14.

Ralf, I know this is probably not the fix you would like to see, any suggestions.

diff -Naur linux-mips-2.6.14.orig/arch/mips/kernel/cpu-probe.c linux-mips-2.6.14/arch/mips/kernel/cpu-probe.c
@@ -121,7 +105,6 @@
    case CPU_24K:
    case CPU_25KF:
    case CPU_34K:
-     case CPU_PR4450:
        cpu_wait = r4k_wait;
        printk(" available.\n");
        break;

@@ -147,6 +130,58 @@
    check_wait();

}

+#ifdef CONFIG_64BIT
+
+/*
+ * On RM5230/5231 all accesses to XKPHYS by LL(D) are forced
+ * to be uncached, bits 61-59 of the address are ignored.
+ *
+ * Apparently fixed on RM5230A/5231A.
+ */
+static inline int check_lld(void)
+{
+ unsigned long flags, value, match, phys, *addr;
+
+ printk("Checking for lld bug... ");
+
+ /* hope the stack is in the low 512MB */
+ phys = CPHYSADDR((unsigned long) &value);
+

+    /* write value to memory */
+    value = 0xfedcba9876543210;
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_UNCACHED, phys);
+    *addr = value;

+
+ /* stop spurious flushes */
+ local_irq_save(flags);
+
+ /* flip cached value */
+ value = ~value;
+

+    /* read value, supposedly from cache */
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_NONCOHERENT, phys);
+    asm volatile("lld %0, %1" : "=r" (match) : "m" (*addr));

+
+ local_irq_restore(flags);
+
+ match ^= value;
+
+ switch ((long) match) {
+ case 0:
+ printk("no.\n");
+ break;
+ case -1:
+ printk("yes.\n");
+ break;
+ default:
+ printk("yikes yes! (%lx/%lx@%p)\nPlease report to <[email protected]>.", value, match, &value);
+ }
+
+ return !match;
+}
+
+#endif
+
/*
* Probe whether cpu has config register by trying to play with
* alternate cache bit and see whether it matters.
@@ -285,8 +320,7 @@

    case PRID_IMP_R4600:
        c->cputype = CPU_R4600;
        c->isa_level = MIPS_CPU_ISA_III;
-        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-                 MIPS_CPU_LLSC;
+        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_LLSC;
        c->tlbsize = 48;
        break;
    #if 0
@@ -366,7 +400,11 @@
        c->cputype = CPU_NEVADA;
        c->isa_level = MIPS_CPU_ISA_IV;
        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |

- MIPS_CPU_DIVEC | MIPS_CPU_LLSC;
+ MIPS_CPU_DIVEC;
+#ifdef CONFIG_64BIT
+ if (check_lld())
+#endif

+            c->options |= MIPS_CPU_LLSC;
        c->tlbsize = 48;
        break;
    case PRID_IMP_R6000:

-- 
----
Jim Gifford
[email protected]

I'll give it a shot and send back results and a updated patch.

One more question, what is the future off the iomap.c file, I didn't include it in my patch. Could it be simply be added to arch/mips/cobalt? I can only speak for the RaQ2, but is it required for any of the other MIPS based machines?

----
Jim Gifford
[email protected]

No. It's broken for machines with multiple PCI busses and I've explicitly rejected the patch which is in kernel.org before.

Ralf

ralf> No. It's broken for machines with multiple PCI busses and I've ralf> explicitly rejected the patch which is in kernel.org before.

Could you explain a bit _how_ broken current kernel.org's arch/mips/lib/iomap.c ? Is it a single mips_io_port_base issue?

I suppose it works as well as traditional way (request_region + in[bwl] for IO resource, request_mem_region + iomap + read[bwl] for MEM resource).

I think it is better than generic iomap.c (except that ioremap_cacheable_cow which is not available for R3000 is used).

---
Atsushi Nemoto