ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

I have a via mini-itx epia 5000 motherboard as a firewall.
It has an ide device: PQI 128MB flash DOM [1] which just plugs directly into the ide connector on the motherboard. 2.6.22-git17 works, although it gives some warning
hda: set_drive_speed_status: status=0x51 { DriveReady SeekComplete Error }
hda: set_drive_speed_status: error=0x04 { DriveStatusError } Full dmesg output [2]

When i compiled any 2.6.23-rc1 kernel out so far it always froze on me. Yesterday i tried 2.6.23-rc1-git3 and i booted until:

VP_IDE: VIA vt8231 (rev 10) IDE UDMA100 controller on pci0000:00:11.1

    ide0: BM-DMA at 0xd000-0xd007, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:pio, hdd:pio

Marking TSC unstable due to: possible TSC halt in C2.
Time: acpi_pm clocksource has been installed.
hda: PQI IDE DiskOnModule, ATA DISK drive
hda: IRQ probe failed (0xfff2)
Clocksource tsc unstable (delta = 84358771493 ns)

So i went back to 2.6.22-git17.
This morning i saw a fresh git4 and compiled/installed that. This kernel actually booted! (once)
I had a netconsole running to catch the lucky event [3]

After about 2 minutes of working however, the whole machine froze. No message at the console, not able to toggle numlock, no magic sysrq key features: solid frozen.
After the power cycle i was not able to boot the same kernel ever again. It was just like a timer had run out and it wouldn't work anymore.

I know my hardware is "ancient" and the flash thing is probably not current. My firewall however is solar powerpowered (10 watt power consumption) and it would be fun if i could run the latest/ greatest kernels on it.

Config files and hdparm info on the DOM in the same directory as dmesg output.

Any comments appriciated.

Danny

[1] http://www.memorydepot.com/ssd_diskonmodule.asp
[2] http://www.dth.net/kernel/via_output_2.6.22_git17
[3] http://www.dth.net/kernel/via_output_2.6.23-rc1-git4

Hi,

Should be harmless for now but we would like to fix it the long-term, please send "hdparm --Istdout /dev/hda" output.

No IDE changes yet after 2.6.22-git17 so something else must have broke down.

Please retry with the latest -git kernel and if the problem is still there install git, get kernel tree and run git-bisect.

Since 2.6.22-git17 works fine the initial "good" commit would be e51f802babc5e368c60fbfd08c6c11269c9253b0 and the initial "bad" one f695baf2df9e0413d3521661070103711545207a (for 2.6.23-rc1).

If you need some practical examples on using git-bisect, this one

http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find-buggy-kernel-patches/

is a very good one (IMO).

This message would indicate broken IRQ routing, however it is no longer present in the log for the kernel 2.6.23-rc1-git4 ([3]).

Interesting that the kernel 2.6.22-git17 (log [2]) doesn't use via82cxxx IDE host driver while the kernel 2.6.23-rc1-git4 (log [3]) does...?

Neat. :)

With the buggy commit number obtained from git-bisect it should be fixed quite quickly.

Thanks,
Bart

I allready made that available in the same subdir:

http://www.dth.net/kernel/

Output repeated here:

voyage:~# hdparm -I /dev/hda

/dev/hda:

ATA device, with non-removable media
Model Number: PQI IDE DiskOnModule
Serial Number:
Firmware Revision: db01.20a
Standards:
Likely used: 1
Configuration:
hard sectored
not MFM encoded
head switch time > 15us
fixed drive
disk xfer rate > 5Mbs
Logical max current
cylinders 500 500
heads 16 16
sectors/track 32 32
--
bytes/track: 0 bytes/sector: 528
CHS current addressable sectors: 256000
LBA user addressable sectors: 256000
device size with M = 1024*1024: 125 MBytes
device size with M = 1000*1000: 131 MBytes
Capabilities:
LBA, IORDY not likely
Buffer type: 0002: dual port, multi-sector
Buffer size: 1.0kB bytes avail on r/w long: 4
Cannot perform double-word IO
R/W multiple sector transfer: Max = 1 Current = 0
DMA: not supported
PIO: pio0 pio1 pio2

voyage:~# hdparm -tT /dev/hda
/dev/hda:
 Timing cached reads:   116 MB in  2.00 seconds =  57.92 MB/sec
 Timing buffered disk reads:   16 MB in  3.01 seconds =   5.32 MB/sec
# hdparm --Istdout /dev/hda

/dev/hda:
045a 01f4 0000 0010 0000 0210 0020 0003 e800 0000 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 0002 0002 0004 6462 3031 2e32 3061 5051 4920 4944 4520 4469 736b 4f6e 4d6f 6475 6c65 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 0001 0000 0200 0000 0200 0000 0001 01f4 0010 0020 e800 0003 0100 e800 0003 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

darn....

I also tried with libata driver on the 2.6.22 series but that wouldn't work as well. I contacted Alan Cox and he made some patches/suggestions that made 2.6.22-git16 with libata work!
I hope his patch will make it into mainline soon.

For the dmesg output of this kernel
http://www.dth.net/kernel/via_output_2.6.22-git16-libata_fix_alan_cox

I will certainly will read/try that.

i booted with routeirq=pci but gave the same negative result.

kernel configs are also in above mentioned webdir

basically copied the 2.6.22-git17 and ran a make oldconfig.

Did a diff -u on those config files and there is really a lot different:

[snip]
-# ATA/IDE/MFM/RLL support
+# Device Drivers
#
[snip]
#
# IDE chipset support/bugfixes
#
+# CONFIG_BLK_DEV_TRM290 is not set
CONFIG_BLK_DEV_VIA82CXXX=y
-# CONFIG_IDE_CHIPSETS is not set -# CONFIG_IDEDMA_AUTO is not set +# CONFIG_BLK_DEV_TC86C001 is not set +# CONFIG_IDE_ARM is not set +CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
-# CONFIG_DMA_NONPCI is not set

I will run a make menuconfig and see if i can clean things up.

Thanks for your feedback/enthousiasm!

Danny

I ran over "make menuconfig" and did a few changes.

http://www.dth.net/kernel/config-2.6.23-rc1-git5

It boots, but freezes solid before doing any "work" as a firewall. I was able to catch boot messages with netconsole

http://www.dth.net/kernel/via_output_2.6.23-rc1-git5

It didn't respond after the last line.
magig sysrq etc, all nada.

Will start to get acquainted with git ;-)

Danny

Maybe try to :

disable BSG ( maybe some leftover bug )
boot acpi=off ( that got merged kind late )

Gabriel

My first git disected kernel wouldn't boot, but with acpi=off it would indeed boot!

As did the 2.6.23-rc1-git5 kernel...

I will bisect further to find out exactly what patch is playing up in my particular setup.

thanks for the tip! ;-)

Danny

Danny ter Haar wrote:
[ added linux-acpi and Len to CC ]

Now while we think is ACPI this should be easy for you to bisect.

This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
merged ACPI so this one should be your first bad one.

Maybe Len has some idea and you don't need to bisect :)

You are welcome :)

Gabriel

Thanks to personal coaching of Gabriel i bisected the last few days.

It looked like this was the cullprit:

22aadf8a07067644e101267ed5003043f2ad05bf is first bad commit

2.6.23-rc1-git[1-6] all lockup solid after either direct or within a couple of minutes (less than 2) after reboot.

They all run fine with "acpi=off" as boot argument.

However, i'm currently running 2.6.23-rc1-git6 with this reverted:

index 18c8b67..6f846be 100644 (file)
@@ -665,8 +665,8 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
        data->max_freq = perf->states[0].core_frequency * 1000;
        /* table init */
        for (i=0; i<perf->state_count; i++) {
-               if (i>0 && perf->states[i].core_frequency ==
-                   perf->states[i-1].core_frequency)
+               if (i>0 && perf->states[i].core_frequency >=
+                   data->freq_table[valid_states-1].frequency / 1000)
                        continue;
 
-----------

I could hardly believe this could be the cause.
And indeed after about 16 minutes the kernel froze again, though longer uptime than other kernels :-(

This hardware is a via epia 5000 with latest bios available (2.07)

It's remarkable that with acpi=off the machine is rocksolid. As is with 2.6.22* kernels _with_ acpi enabled! weeks of uptime before i wanted to upgrade to a new kernel ;-)

I'm thinking of redoing the git disect but this time really powercycle the unit between kernels since it "seems/feels" like a timer which really counts to 0 and then locks the machine.
After a lockup/freeze i cant boot the kernel another time with acpi
enabled: it will simply hang after booting init.
And again nobody home: no keyboard activity whatsoever.

I've put the dmesg of 2.6.23-rc1-git6 with and without acpi=off online:
http://www.dth.net/kernel/via_output_2.6.23-rc1-git6-acpi_off
http://www.dth.net/kernel/via_output_2.6.23-rc1-git6-acpi_on

I would like to hear if someone has an idea how to tackle this problem.

Danny

Please attach the output from acpidump to
http://bugzilla.kernel.org/show_bug.cgi?id=7880
We'll likely be able to tell from it if that patch has any real effect on your system, or if you test result was from some unrelated event.

Also, please test with CONFIG_X86_ACPI_CPUFREQ=n to remove the acpi-cpufreq driver (and thus this patch) from your kernel. If it still fails, then we know that this driver (and this patch) are not related to the failure.

Hmmm, okay, the "big hammer" works. Please see if any of these smaller hammers work:

pnpacpi=off
acpi=noirq
notsc

thanks,
-Len

done
also on : http://www.dth.net/kernel/acpidump_output_via_c3_5000

Reverting the patch didn't make a real difference, it only took longer (eg 16 minutes before lockup in stead of 2-3 minutes)

Will do.
This machine(firewall) is solar powered so i tried to let it consume as less power as possible
This is in my startup script at the moment:

echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

More testing to do, however, sunday i'm going abroad for a while, all stuff will be in storage. If i cant find in the next few days (packing everything in moving boxes) it might take a while before i can test again. Sorry for that.

Danny

Went out for dinner, machine was frozen on return. One less on the checklist ..

died within 2 minutes

After 36 minutes

It wasn't enabled in any of the kernels i ran:

# grep X86_ACPI ../configs/config*
../configs/config-2.6.22-git14-c3-via5000-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.22-git17-c3-via5000-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git5:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git6:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git9:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-mm1-c3-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set

_none_ of these options will work. The machine locks up solid. And that's what i'm the most surprised about. The last time i withnessed a solid kernel lockup without any panic/notification is _years_ ago.

I also tried a kernel without "CPU Frequency scaling", so it is not the scaling up/down in Mhz of the cpu.
I've even just compiled/run a kernel with libata and even that one still locks up.

New round of git bisects i guess (this time powercycle instead of reboot and wait few hours before ginving it "bisect good" )

Will report back.

Danny

Okay

Redid git bisect from start:

git bisect log
git-bisect start
# good: [098fd16f00005f665d3baa7e682d8cb3d7c0fe6f] Linux 2.6.22 git-bisect good 098fd16f00005f665d3baa7e682d8cb3d7c0fe6f
# good: [098fd16f00005f665d3baa7e682d8cb3d7c0fe6f] Linux 2.6.22 git-bisect good 098fd16f00005f665d3baa7e682d8cb3d7c0fe6f
# bad: [7d57c74238cdf570bca20b711b2c0b31a553c1e5] Linux 2.6.23-rc1 git-bisect bad 7d57c74238cdf570bca20b711b2c0b31a553c1e5
# good: [1f1c2881f673671539b25686df463518d69c4649] Merge branch # 'upstream-linus' of
# master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/netdev-2.6 git-bisect good 1f1c2881f673671539b25686df463518d69c4649
# good: [5b7f13bd26a0c1d394a1a1f2bb6de5130c3a3843] coda: update module

# information
git-bisect good 5b7f13bd26a0c1d394a1a1f2bb6de5130c3a3843
# good: [74889e41d9a1f80928130a02af9b010673bc5ba7] [CELL] cbe_cpufreq:
# reorganize code
git-bisect good 74889e41d9a1f80928130a02af9b010673bc5ba7
# good: [bdda1561ffda764583a295229db66d94cf6038a3] x86_64: x86_64 - Use
# non locked version for local_cmpxchg()
git-bisect good bdda1561ffda764583a295229db66d94cf6038a3
# bad: [d7f5e3df3574c6e38b99f5fe22f15540b2b9811d] Merge branch
# 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
git-bisect bad d7f5e3df3574c6e38b99f5fe22f15540b2b9811d
# bad: [08e31686d6d119ba26bf0690f5f872f6f5bd1a97] Pull thinkpad into
# release branch
git-bisect bad 08e31686d6d119ba26bf0690f5f872f6f5bd1a97
# bad: [f79e3185dd0f8650022518d7624c876d8929061b] Pull misc into release
# branch
git-bisect bad f79e3185dd0f8650022518d7624c876d8929061b
# good: [22aadf8a07067644e101267ed5003043f2ad05bf] Pull bugzilla-7880
# into release branch
git-bisect good 22aadf8a07067644e101267ed5003043f2ad05bf
# good: [e8b495fe09bc793ae26774e7b2667f7f658d56e2] Pull dock-bay into
# release branch
git-bisect good e8b495fe09bc793ae26774e7b2667f7f658d56e2
# good: [18eab8550397f1f3d4b8b2c5257c88dae25d58ed] ACPI: Enable C3 even
# when PM2_control is zero
git-bisect good 18eab8550397f1f3d4b8b2c5257c88dae25d58ed
# good: [7a9b9068b8c7b526894904c0b9b90354825b8fbd] ACPI: update
# feature-removal-schedule.txt, /sys/firmware/acpi/namespace is gone
git-bisect good 7a9b9068b8c7b526894904c0b9b90354825b8fbd
# good: [8b8eb7d8cfc6cd95ed00cd58754e8493322505bd] ACPI: update ACPI
# proc I/F removal schedule
git-bisect good 8b8eb7d8cfc6cd95ed00cd58754e8493322505bd

f79e3185dd0f8650022518d7624c876d8929061b is first bad commit

I will store this particular firewall setup friday (will go into storage for 2 months), if somebody would like more info, me to try something, please send it to me asap.

Thanks for looking at this.

Danny