Skip to contentSkip to conversation

Preempt problems

Linux-ia64 Peter Chubb 5 participants 8 messages Feb 3, 2003 – Feb 14, 2003

lore
Share & tools
lore
#0 lore raw

Hi David,
Just a heads up --- there are still two major problems with the
patch I submitted for preemption support, so I suggest you don't merge it yet (although I'd very much appreciate having it reviewed and tested by third parties).

The two problems I see are:
-- perfmon missing interrupts *even with CONFIG_PREEMPT turned
off!*
-- weird kernel page faults apparently during ia64_leave_kernel()
(from the stack backtrace), but with an IP address in region 0, after some variable amount of uptime, apparently independent of anything happening on the machine (at least, I haven't yet been able to isolate anything that causes it).

--
Dr Peter Chubb [email protected]
You are lost in a maze of BitKeeper repositories, all almost the same.

#1 lore raw

Peter,

Quoted 10 lines · Peter Chubb

Hi David,
Just a heads up --- there are still two major problems with the
patch I submitted for preemption support, so I suggest you don't merge it yet (although I'd very much appreciate having it reviewed and tested by third parties).

The two problems I see are:
-- perfmon missing interrupts *even with CONFIG_PREEMPT turned
off!*

As for perfmon, there are some known issues with perfmon and the O(1) scheduler (deadlocks during ctxsw in SMP). I am not sure it affects your particular test case. I had postponed fixing this because I am working on a new perfmon code base for 2.5 in which (hopefully) all problems are gone. However a somewhat related issue came up last week and I decided to fix some of the problems. I will try to give a new patch to David this week.

As for preemption and perfmon, I haven't had a chance to look at the patch yet. There are some assumptions about not being preemptable at several places.

Signature · 1 line
-Stephane
#2 lore raw
In reply to Peter Chubb
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Peter" = Peter Chubb <[email protected]> writes:

Peter> * -- weird kernel page faults
Peter> apparently during ia64_leave_kernel() (from the stack Peter> backtrace), but with an IP address in region 0, after some Peter> variable amount of uptime, apparently independent of anything Peter> happening on the machine (at least, I haven't yet been able to Peter> isolate anything that causes it).

I just got one of these on a straight 2.5.59+davidm's patches, on a vanilla zx2000. The kernel address is always the same, at 0xffe6bf10, regardless of the kernel, and regardless of the command that was (apparently) running at the time.

I've appended the Oops message. On a previous (different kernel) I
printed out the iipa: e00000003ff04420cc1
It also appears bogus. Note that interrupts are off in the psr.

I'm at a loss as to how to debug this thing now. Can anyone suggest anything? It *could* be a hardware problem (I've had problems with this machine before).

Unable to handle kernel paging request at virtual address 00000000ffe6bf10

automount[288]: Oops 4294967296

Pid: 288, CPU 0, comm: automount
psr : 0000101008022018 ifs : 800000000000048d ip : [<00000000ffe6bf10>] Not tainted
ip is at 0xffe6bf10
unat: 0000000000000000 pfs : 0000000000000d22 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 769dbd656f5a5865
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f b0 : e00000003ff04430 b6 : e00000003fef6a80 b7 : e00000003fef37b0 f6 : 1003e0000000000003fff f7 : 1003ecccccccccccccccd f8 : 1000cfffc000000000000 f9 : 1003e0000000000000008 r1 : e00000003fe40000 r2 : e00000003ff04430 r3 : 0000000000000000 r8 : 00000000ffe6bf10 r9 : 0000000000000000 r10 : 0000000000000000 r11 : 0000000000000000 r12 : e0000040428d7380 r13 : e0000040428d0000 r14 : c0000000ff4661f8 r15 : c0000000ff44a040 r16 : 000000000074a040 r17 : 0000000000000001 r18 : 00000000000000ff r19 : 0000000000000200 r20 : 00000000000002ff r21 : a0000000000100c8 r22 : e00000003fef37b0 r23 : e000000004ab8318 r24 : e0000000049eec10 r25 : e000000004aa4228 r26 : 0000000000000006 r27 : e000000004ab93e8 r28 : 0000000000000019 r29 : 0000000000000000 r30 : 0000000000000000 r31 : 0000000000000000

Call Trace:
 [<e000000004416560>] show_stack+0x80/0xa0 sp=0xe0000040428d6f90 bsp=0xe0000040428d1b58
 [<e00000000442cef0>] die+0x110/0x1a0 sp=0xe0000040428d7150 bsp=0xe0000040428d1b30
 [<e000000004448450>] ia64_do_page_fault+0x310/0x840 sp=0xe0000040428d7150 bsp=0xe0000040428d1ad0
 [<e000000004411520>] ia64_leave_kernel+0x0/0x240 sp=0xe0000040428d71e0 bsp=0xe0000040428d1ad0
 <0>Kernel panic: Aiee, killing interrupt handler!

In interrupt handler - not syncing

#3 lore raw
In reply to Peter Chubb
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Peter Chubb

Peter> I just got one of these on a straight 2.5.59+davidm's Peter> patches, on a vanilla zx2000. The kernel address is always Peter> the same, at 0xffe6bf10, regardless of the kernel, and Peter> regardless of the command that was (apparently) running at Peter> the time.

Can't say I have ever seen anything like this. It does look like the top 32 bits somehow get truncated to zero. Rather weird.

Peter> I'm at a loss as to how to debug this thing now. Can anyone Peter> suggest anything? It *could* be a hardware problem (I've had Peter> problems with this machine before).

If you had problems before, a hw problem sounds like a real possibility. If I can find some time, I'll try your patch on some of our machines.

	--david
#4 lore raw
In reply to Peter Chubb

Hello,

I've just found a problem when using preemption patch with SMP on a 2.5.59 kernel. (with my version of the _raw_write_trylock() macro)

It seems, that the SCSI driver have some problems (at boot time):
As a result, sometimes, one of my partition (type ext3) can not be mounted.
Here is the only related message in the log file :

Feb  5 17:54:13 zli22 kernel: scsi: Underflow detected - retrying command.

Sometimes, the partition is "half-mounted". I can only read one part of the files. If I try to mount it again or to unmount it, I've got a message saying that the device is busy.

> mount /dev/sda4
umount: /home: device is busy

It seems to me, that it is a deadlock. But it's not easy to identify it because, it doesn't happen every time I boot the machine (about 1/2, with my machines that have 2 cpus))

It could be a problem in the "_raw_write_trylock()", but I have made some tests, and it seems that the macro "returns" the good value.

---------------

Quoted 2 lines

... because the rw_lock value is composed with :
- 1 bit for the write "flag" (the most significant bit of a _long_ value)

Oh... I meant MSB of an _int_ value !

Quoted 1 line

- 31 bits for the read flags

Regards

---------
Joel

#5 lore raw
In reply to Peter Chubb
Quoted 5 lines · Stephane Eranian

Peter,

Quoted 1 line · Peter Chubb

Stephane,

Does the deadlock you describe here look at all like the bug report that Jack Steiner has submitted for our Altix kernel?
(We're using the O(1) scheduler and pfmon.) (It certainly sound similar.) Details attached.

Is anyone working this issue that you know of?

Quoted 18 lines

As for perfmon, there are some known issues with perfmon and the O(1) scheduler (deadlocks during ctxsw in SMP). I am not sure it affects your particular test case. I had postponed fixing this because I am working on a new perfmon code base for 2.5 in which (hopefully) all problems are gone. However a somewhat related issue came up last week and I decided to fix some of the problems. I will try to give a new patch to David this week.

As for preemption and perfmon, I haven't had a chance to look at the patch yet. There are some assumptions about not being preemptable at several places.

--
-Stephane

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

Signature · 9 lines
Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

From: [email protected] ([email protected])
Subject: BUG 881594 - Deadlock in perfmon - pfm_fetch_regs()
To: [email protected], [email protected]
Status:
X-Mozilla-Status: 8001
X-Mozilla-Status2: 00000000
X-UIDL: 3bd895570000681c

View Incident: http://co-op.engr.sgi.com/BugWorks/code/bwxquery.cgi?search=Search&wlong=1&view_type=Bug&wi=881594

Submitter : steiner Submitter Domain : sgi.com
Assigned Engineer : raybry Assigned Engineer Domain : sgi.com
Assigned Group : linux-mckinley Category : software
Reported by Customer : F Priority : 2
Project : snlinux Status : open
Description :
Ferarri hung this morning running a mixture of

        0xe000003068b40000 00024267 00024266  0  006  stop  0xe000003068b407d0 code3
        0xe00000305c608000 00024273 00024250  0  000  stop  0xe00000305c6087d0 pfmon
        0xe00000300c738000 00024275 00010688  0  000  stop  0xe00000300c7387d0 go.bottle
        0xe000003051018000 00024278 00010688  0  000  stop  0xe0000030510187d0 go.bottle
        0xe000003020058000 00024281 00010688  0  000  stop  0xe0000030200587d0 go.bottle
        0xe00000306c6e8000 00024284 00010688  0  006  stop  0xe00000306c6e87d0 go.bottle
        0xe000003049098000 00024287 00010688  0  000  stop  0xe0000030490987d0 go.bottle
        0xe000003021558000 00024274 00024273  0  000  stop  0xe0000030215587d0 code3
        0xe00001b030900000 00024295 00024284  0  006  stop  0xe00001b0309007d0 pfmon
        0xe00001b03ad30000 00024296 00024295  0  005  stop  0xe00001b03ad307d0 code3
        0xe00000302a240000 00024297 00024278  0  000  stop  0xe00000302a2407d0 pfmon
        0xe000003028980000 00024298 00024275  0  000  stop  0xe0000030289807d0 pfmon

From the leds, it appeared that cpu 2 & 3 were hard hung & not processing interrupts. I nmi'ed the system. Cpu 2 was hung here (I think this is right - 90% confidence. Someone reset the system before I finished digging out the info I needed):

     1  99  1  smp_call_function_single
     7  ??  2      pfm_fetch_regs
     8  ??  3          pfm_load_regs
     9  ??  4              ia64_load_extra
    10  ??  5                  __switch_to
    11  ??  6                      switch_to
    12  ??  7                          context_switch
    13  ??  8                              schedule

cpu 2 was in the function smp_call_function_single spinnning with interrupts disabled trying to lock call_lock. Cpu 3 was hung the same way that cpu was hung.

Another cpu was holding the call_lock & was waiting for cpu 2 to respond to an IPI. Since cpu 2 was spinning with interrupts disabled, it was not responding.
The cpu holding the lock was here:

        0xe002000000045af0 smp_call_function+0x470
        0xe002000000045350 smp_flush_tlb_all+0x30
        0xe002000000051550 flush_tlb_range+0x50
        0xe002000000125090 swap_out+0x9f0
        0xe002000000126350 shrink_cache+0xb70
        0xe0020000001269e0 shrink_caches+0x100
        0xe002000000126ad0 try_to_free_pages+0x70
        0xe002000000128b40 balance_classzone+0xe0
        0xe0020000001295c0 __alloc_pages+0x420
        0xe0020000001297c0 __get_free_pages+0xc0
        0xe002000000120260 kmem_cache_grow+0x280
        0xe002000000121580 kmem_cache_alloc+0x460
        0xe00200000033fc60 kmem_zone_zalloc+0xa0
        0xe0020000002d5980 xfs_efd_init+0x80
        0xe00200000030e030 xfs_trans_get_efd+0x30
        0xe00200000029cf40 xfs_bmap_finish+0x1a0
        0xe0020000002e5cb0 xfs_itruncate_finish+0x2d0
        0xe002000000318640 xfs_inactive+0x5e0
        0xe00200000033ea20 vn_rele+0x140
        0xe00200000033c9f0 linvfs_clear_inode+0x30
        0xe0020000001741d0 clear_inode+0x370
        0xe002000000175d70 iput+0x4b0
        0xe002000000170160 d_delete+0x180
        0xe00200000015c690 vfs_unlink+0x650
        0xe00200000015c930 sys_unlink+0x210
        0xe00200000000ea00 ia64_ret_from_syscall

I dont believe that pfm_fetch_regs should be calling smp_call_function_single unless interrupts are enabled.

#6 lore raw
In reply to Peter Chubb

Ray,

Quoted 8 lines · Ray Bryant

Does the deadlock you describe here look at all like the bug report that Jack Steiner has submitted for our Altix kernel?
(We're using the O(1) scheduler and pfmon.) (It certainly sound similar.) Details attached.

Is anyone working this issue that you know of?

This problem was reported to me by Intel and others. It was affecting the 2.5 kernel only. However given that RHAS 2.1 is also using the O(1) scheduler, I had to fix it in 2.4.18/RHAS. I don't exactly know which kernel you are using at SGI. Note that the fix has been propagated to 2.5.60 as well. I can provide the patch to you but note that it also includes updates to perfmon to bring it closer to what we have in 2.4.20. My goal is to minimize the number of variations out there ;->

Signature · 1 line
-Stephane
#7 lore raw
In reply to Peter Chubb

Stephane,

We're using a 2.4.19 kernel with a number of patches and SGI fixes, and some backports from 2.5, as well as our machine
specific stuff. A more complete description of what we run is:

--2.4.19 from www.kernel.org
--IA64 patch
--Atlas patches including discontinguous memory support, VM support --O(1) scheduler -- including some fixes for large CPU counts, etc --SGI machine specific H/W support
--SGI patches and fixes for bugs and scalability
--PAGG
--CSA (Comprehensive System Accounting)
--XSCSI (our version of SCSI support from IRIX -- this is a module/closed source)

Of interest for this discussion is the O(1) scheduler and perfmon.

If you could send me your patch, I will take a look at it and see what applies.

We will eventually be going to RHAS, but at the moment we are running a "free-bytes" clone of RH 7.2.

I understand exactly why you are trying to keep the number of variations small.

Thanks,

Quoted 23 lines · Stephane Eranian

Ray,

Quoted 8 lines · Ray Bryant

Does the deadlock you describe here look at all like the bug report that Jack Steiner has submitted for our Altix kernel?
(We're using the O(1) scheduler and pfmon.) (It certainly sound similar.) Details attached.

Is anyone working this issue that you know of?

This problem was reported to me by Intel and others. It was affecting the 2.5 kernel only. However given that RHAS 2.1 is also using the O(1) scheduler, I had to fix it in 2.4.18/RHAS. I don't exactly know which kernel you are using at SGI. Note that the fix has been propagated to 2.5.60 as well. I can provide the patch to you but note that it also includes updates to perfmon to bring it closer to what we have in 2.4.20. My goal is to minimize the number of variations out there ;->

--
-Stephane

Signature · 9 lines
Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------