Problem: ACPI related soft lockup on Fujitsu-Siemens AMILO Si 2636

Hello

I'm experiencing a problem with my new Fujitsu-Siemens AMILO Si 2636 laptop. When the system is under heavy load it locks up and a message about soft lockup roll on the terminal. The message differs from time to time but it's always a soft lockup caused by kacpid and the associated call trace is always somewhere in the ACPI AML parsing code. I'm guessing there is a bug in the AML parser or more likely in the Amilo's AML code

The lockup only occur if ACPI is active and it seems to be related to the thermal module. Most of the call traces contain the function acpi_thermal_get_temperature but not all of them. I use the most recent 1.0d 25.03.2008 BIOS.

I believe kernel bug #6433 is related to this. The circumstance and call trace looks similar.
http://bugzilla.kernel.org/show_bug.cgi?id=6433

A typical message looks like this:

BUG: soft lockup - CPU#1 stuck for 61s! [kacpid:87]
Modules linked in: i2c_i801
CPU 1:
Modules linked in: i2c_i801
Pid: 87, comm: kacpid Tainted: G   M      2.6.27-rc8 #1
RIP: 0010:[<ffffffff80423809>]  [<ffffffff80423809>] acpi_ns_delete_namespace_by_owner+0x66/0xde
RSP: 0018:ffff88013fa19ce0  EFLAGS: 00000207
RAX: ffff88013f81a820 RBX: ffff88013f81a820 RCX: 0000000000000000
RDX: ffff88013f81a820 RSI: ffff88013f81a700 RDI: ffff88013f81a800
RBP: ffff880139370cf0 R08: 0000000000000000 R09: ffff880139370cf0
R10: ffff88013eaa6430 R11: ffff88013eaa7000 R12: ffff88013eaa6430
R13: ffff88013eaa7000 R14: ffff88013f81a5a0 R15: ffffffff80414149
FS:  0000000000000000(0000) GS:ffff88013f809680(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000004e3fd0 CR3: 000000013e9da000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
[<ffffffff80416ea4>] ? acpi_ds_terminate_control_method+0x6e/0xc1
[<ffffffff8042584c>] ? acpi_ps_parse_aml+0x18c/0x2c8
[<ffffffff80426c47>] ? acpi_ps_execute_method+0x135/0x1e2
[<ffffffff80423a13>] ? acpi_ns_evaluate+0x14f/0x1ac
[<ffffffff80423566>] ? acpi_evaluate_object+0x131/0x1df
[<ffffffff8022efe3>] ? __wake_up+0x43/0x70
[<ffffffff80414d55>] ? acpi_evaluate_integer+0x8c/0xc7
[<ffffffff80438edb>] ? acpi_thermal_get_temperature+0x2e/0x3b
[<ffffffff80439127>] ? acpi_thermal_check+0x84/0x3e2
[<ffffffff804143d2>] ? acpi_os_execute_deferred+0x0/0x2c
[<ffffffff804143f5>] ? acpi_os_execute_deferred+0x23/0x2c
[<ffffffff80249b39>] ? run_workqueue+0x79/0x100
[<ffffffff8024a560>] ? worker_thread+0x0/0x120
[<ffffffff8024a61f>] ? worker_thread+0xbf/0x120
[<ffffffff8024d7a0>] ? autoremove_wake_function+0x0/0x30
[<ffffffff8024a560>] ? worker_thread+0x0/0x120
[<ffffffff8024d38b>] ? kthread+0x4b/0x80
[<ffffffff8020c629>] ? child_rip+0xa/0x11
[<ffffffff8024d340>] ? kthread+0x0/0x80
[<ffffffff8020c61f>] ? child_rip+0x0/0x11

Environment:
I run gentoo with the 2.6.27-rc8 kernel and I have also tried the 2.6.25 and 2.6.24 kernels using 32 and 64-bit OS with the same result.

Steps to reproduce:
Boot a recent kernel on a Fujitsu-Siemens AMILO Si 2636 laptop and put the system under heavy load. A kernel make -j3 works but a make -j1 is not enough. Wait 5-10 min until lockup.

Outputs:
The output of useful programs including acpidump can be viewed here http://www.cyd.liu.se/~tholi945/acpi-bug-2008-10-06/

/Thomas Lindroth

I've done some debugging of this problem myself and reached some conclusions.

I tried activating the ACPI_LV_INFO ACPI debug mode and observed what happens when the system is under load. I saved one trace and put it in the file acpi_debug_log at http://www.cyd.liu.se/~tholi945/acpi-bug-2008-10-06/

When the temp goes over the ACPI passive trip point the processor gets throttled. It usually never goes over T4 before the temp falls below the passive temp but sometimes it reaches the highest T7. If it reach T7 the next call to _TMP or _L18 never returns. I've confirmed this by running echo T7 > /proc/acpi/processor/CPU*/throttling and then tried to read from /proc/acpi/thermal_zone/TZ00/temperature and it fails in the same way.

I've tried activating the ACPI_LV_PARSE ACPI debug mode, enter T7 and read from temp. I put the result of that in the trace_debug file at the same adress.

The AML call chain looks like this _TMP -> PMRD -> RCMD -> WIBF WIBF always returns 1 causing the interpreter to get stuck in the while loop in PMRD.

While (RCMD (0x80, Local0))

{
    Noop
    Noop
    Store (PMUC, Local5)
    If (And (Local5, One, Local2))
    {
        Store (PMUD, Local5)
    }
}

I don't really understand how AML code works so I can't get much further than this.

I did not experience complete lockups in Windows but I did notice some minor stalls. Maybe the Windows AML parser can break infinite loops or maybe the stalls is a normal part of the Windows experience. I tried to boot with the acpi_osi=Linux option active but it didn't make any difference.

/Thomas Lindroth

This is AML implementation of EC driver... it relies on certain speed of AML interpreter to work properly (look for Noop inserted as delay).

It is possible to replace this driver with the native Linux one, which might be more robust, but this involves heavy DSDT hacking.

Overall, it is heavily mis-designed BIOS, so don't expect it to be fixed on Linux side.

Regards,
Alex.

Please check if making Noop a small delay helps

index 6a81c44..acd5fb6 100644
@@ -1368,6 +1368,7 @@ acpi_ds_exec_end_control_op(struct acpi_walk_state * walk_state,
 	case AML_NOOP_OP:
 
 		/* Just do nothing! */
+		udelay(100);
 		break;
 
 	case AML_BREAK_POINT_OP:

I've tried the patch with an udelay of 100 but it did not make any difference. I've also tried raising it to 200 but that created an unbootable system because one of the ACPI init methods failed.

You are right, this is a really broken BIOS.

I find it odd that only I experience this problem. According to one site this laptop is the 7:th most popular laptop right now so someone else should have reported something.

/Thomas Lindroth

Could you provide a short explanation what should be fixed in which way in the BIOS tables. Best in a way that I can just forward this.
It's hard in the laptop area to convince fsc to fix it, but I at least like to try and to tell them.
Hmm, I think the last two mails should already be enough, I think I got it.

Is there a general hint I could add into the ACPI Linux guideline, like:
Don't implement EC data/index register reads/writes in AML yourself. Define a EC variable and e.g. use Store(ECXY, Local0) to read and Store(Local0, ECXY) to write to specific EC bytes.

Did I get this right?

Thanks,

Thomas

T7 is not very popular setting :)

As I understand, this is not FSC, but FIC outsource. There is exactly same thing happening in new FIC laptop.

Declare EC interface might be a better wording.

Yes.
My understanding is that Windows does not support 2 EC devices, so these people need to implement second in AML.
So, it might be worth noting that Linux can handle any number of EC devices.

Regards,
Alex.

...

Can these guys do it right if we return true for:
_OSI("Multiple EC capable")

How does a second EC declaration look like?
Just a second EC OperationRegion?
Ahh, it's a whole device, it would still work. It's just the first time that the device is only used on Linux and not on Windows.
This would also be declared in Windows, because you cannot set conditions around declarations, but it would not hurt:
This is done in _OSI init section:

                    If (\_OSI ("Multiple EC capable"))
                    {
                        Store (0x01, \MIEC)
                    }

Somewhere in the Scope you have this unconditionally. Arghh, that would certainly not work on Windows then or could at least be a potential risk there?:

                    /* Second EC, unused on Windows */
                    Device (EC2)
                   {
                        Name (_HID, EisaId ("PNP0C09"))
                        OperationRegion (ECOR, EmbeddedControl, 0x00, 0x0100)
                        Field (ECOR, ByteAcc, NoLock, Preserve)
                        {
                            RTMP,   1,
                            ...
                        }
                   }

Where the actual value is read:

/* Linux case */
if(LEqual(MIEC, 1))
Store(RTMP, 1)

/* Windows case */

else {
    While (RCMD (0x80, Local0))
    {
        Noop
        Noop
        Store (PMUC, Local5)
        If (And (Local5, One, Local2))
        {
            Store (PMUD, Local5)
        }
    }
}

---------------
Better ideas?
The problem is the second EC HID device which cannot be hidden on Windows... If Windows is really too dumb for two ECs and vendors are therefore really doing such crap as above, we should think about alternatives..., the EC driver simulation looks rather unsupportable (It misses some locks, but not in AML namespace, but the ones in drivers/acpi/ec.c, right?).

IIRC the ACPI spec explicitly mentions a two (or multiple?) EC PC configuration somewhere? Anyway...

Thomas

please send the output from

grep . /proc/acpi/thermal_zone/*/*

from the log, it looks like your _PSV returns 3520, or 352.0K or 78C and that is reasonable, but note that one option you have to workaround is to either disable or that trip point via

thermal.psv=-1
or
thermal.psv=100
for example.

But the bigger question is why the temperature is hitting 78 -- which is quite hot.
Grep above will tell us if there is ACPI fan control, but I don't see any in the logs. Can you hear the fans spinning? Do they go faster when the system heats up?
do they work better after you blow the dust our of the fan?

-Len

/proc/acpi/thermal_zone/TZ00/cooling_mode:<setting not supported> /proc/acpi/thermal_zone/TZ00/polling_frequency:<polling disabled>
/proc/acpi/thermal_zone/TZ00/state:state: ok

/proc/acpi/thermal_zone/TZ00/temperature:temperature:             53 C
/proc/acpi/thermal_zone/TZ00/trip_points:critical (S5):           94 C
/proc/acpi/thermal_zone/TZ00/trip_points:passive:                 86 C: tc1=0 tc2=10 tsp=50 devices=CPU0 CPU1
/proc/acpi/thermal_zone/TZ01/cooling_mode:0 - Active; 1 - Passive
/proc/acpi/thermal_zone/TZ01/polling_frequency:<polling disabled>
/proc/acpi/thermal_zone/TZ01/state:state:                   ok
/proc/acpi/thermal_zone/TZ01/temperature:temperature:             27 C
/proc/acpi/thermal_zone/TZ01/trip_points:critical (S5):           94 C
/proc/acpi/thermal_zone/TZ01/trip_points:passive:                 78 C: tc1=0 tc2=10 tsp=2 devices=CPU0 CPU1
/proc/acpi/thermal_zone/TZ01/trip_points:active[0]:               80 C: devices=FAN1
/proc/acpi/thermal_zone/TZ01/trip_points:active[1]:               67 C: devices=FAN0
/proc/acpi/thermal_zone/TZ01/trip_points:active[2]:               55 C: devices=FAN2

I'm not sure why there are two TZ for the CPUs.

TZ00._PSV returns 3520 or 3590 depending on if the temp is above or below the previous _PSV value.

The fans work and spin up under load. TZ01 got fan control but I never see anything about TZ01 in the logs so I guess the fans are controlled by the BIOS. The computer is new and not dusty.

/Thomas Lindroth