Skip to contentSkip to conversation

system hangs momentarily every 4 mins 2 secs

DISCUSSION tom arnall 5 participants 11 messages Aug 26, 2006 – Sep 3, 2006 6 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1155859868.6787.8.camel@localhost

tom arnall

author
#3 lore raw

My system hangs momentarily every 4 mins 2 secs. The event coincides with a 'top' entry in which 'events/0' grabs 50-95% of cpu capacity. How can I determine what is causing the event?

Thanks,

Tom Arnall
north spit, ca

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

#4 lore raw
Quoted 3 lines · On Sat, 26 Aug 2006 12:55:33 -0700 tom arnall <[email protected]> wrote:

My system hangs momentarily every 4 mins 2 secs. The event coincides with a 'top' entry in which 'events/0' grabs 50-95% of cpu capacity. How can I determine what is causing the event?

First of all, I'm a newbie. But I'm interested in your subject. What about trying 'pstree'

$ pstree -p
<example output>
 |--events/0(3)--|-aio/0(44)
                 |-kacpid(5)
                 |-kblockd/0(31)
                 |-khelper(4)
                 |-pdflush(41)
                 |-pdflush(42)
<example output>

Then take the pid's and use top to monitor them.

$ top -p3,44,5,31,4,41,42

I don't know if the subtrees will show you anything, but maybe worth a try.

Mike

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

#5 lore raw

Hi,

Quoted 6 lines · On 8/27/06, [email protected] <[email protected]> wrote:
Quoted 3 lines · On Sat, 26 Aug 2006 12:55:33 -0700 tom arnall <[email protected]> wrote:

My system hangs momentarily every 4 mins 2 secs. The event coincides with a 'top' entry in which 'events/0' grabs 50-95% of cpu capacity. How can I determine what is causing the event?

This is a long-shot but the last time I had the same problem it turned out that there had developed a loose connection with the heat-sink and processor on the motherboard.
As a result, the processor used to over-heat in about every 3 minutes or so, the watchdogs would come into picture and cause the machine to hang or reboot.... usually reboot.

Just give a quick check on your hardware before probing into the software side.

Signature · 8 lines
Raseel.
http://osd.byethost8.com
http://raseel.livejournal.com
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs
author
#6 lore raw
Quoted 17 lines · Raseel Bhagat

Hi,

Quoted 6 lines · On 8/27/06, [email protected] <[email protected]> wrote:

On Sat, 26 Aug 2006 12:55:33 -0700

Quoted 3 lines · tom arnall

My system hangs momentarily every 4 mins 2 secs. The event coincides with a 'top' entry in which 'events/0' grabs 50-95% of cpu capacity. How can I determine what is causing the event?

This is a long-shot but the last time I had the same problem it turned out that there had developed a loose connection with the heat-sink and processor on the motherboard.
As a result, the processor used to over-heat in about every 3 minutes or so, the watchdogs would come into picture and cause the machine to hang or reboot.... usually reboot.

Quoted 2 lines

Just give a quick check on your hardware before probing into the software side.

how did you identify the problem with the heat sink?

is it likely that a hardware problem would cause the hanging phenomenon at such a precise interval? i ran 'top' in batch mode to a file for ~.5 hour and also wrote down the time whenever the hang occurred, then i compared the output of top with the 'hang moments'. there was an exact coincidence with 'top' entries in which 'events/0' was gobbling the cpu. and these events occurred exactly 4 mins 2 secs from each other.

i also logged the content of /proc/interrupts at intervals of ~.2 sec's. in this case, there was no change at the 'hang moments' except that i got only 1 sample for the second in which the hang occurred. in all other cases, of course, i got 5 samples per second.

thanks in advance,

tom arnall
north spit, ca

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

#7 lore raw
Quoted 27 lines · tom arnall
Quoted 19 lines · Raseel Bhagat

Hi,

Quoted 7 lines · On 8/27/06, [email protected] <[email protected]> wrote:

On Sat, 26 Aug 2006 12:55:33 -0700

Quoted 3 lines · tom arnall

My system hangs momentarily every 4 mins 2 secs. The event coincides with a 'top' entry in which 'events/0' grabs 50-95% of cpu capacity. How can I determine what is causing the event?

This is a long-shot but the last time I had the same problem it turned out that there had developed a loose connection with the heat-sink and processor on the motherboard.
As a result, the processor used to over-heat in about every 3 minutes or so, the watchdogs would come into picture and cause the machine to hang or reboot.... usually reboot.

Quoted 2 lines

Just give a quick check on your hardware before probing into the software side.

how did you identify the problem with the heat sink?

Quoted 8 lines

is it likely that a hardware problem would cause the hanging phenomenon at such a precise interval? i ran 'top' in batch mode to a file for ~.5 hour and also wrote down the time whenever the hang occurred, then i compared the output of top with the 'hang moments'. there was an exact coincidence with 'top' entries in which 'events/0' was gobbling the cpu. and these events occurred exactly 4 mins 2 secs from each other.

i also logged the content of /proc/interrupts at intervals of ~.2 sec's. in this case, there was no change at the 'hang moments' except that i got only 1 sample for the second in which the hang occurred. in all other cases, of course, i got 5 samples per second.

thanks in advance,

tom arnall
north spit, ca

Tom -- You are right to be skeptical about the heat-sink suggestion. I've had problems similar to that one, and they ALWAYS cause either a reboot or a permanent hang (possibly with a kernel OOPS). They neve cause the sort of transient problem you describe.

What you are seeing isn't really a "hang"; it is just the system becoming busy with a task other than the one currently onscreen. The 2 seconds part intrigues me; might it be the case that the "hang" lasts for 2 seconds? Or might the clock be 2 seconds off after it occurs? (Hard to check from one instance, but over a day, this would lead to a 6-minute clock drift, easy to spot.)

I'm pretty sure "events" are what older kernels identify as keventd, a process internal to the kernel. That is, you're seeing a line in top something like this one ...

4 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/0

... except that the %CPU column is a lot higher.

If I'm right, and if you're using a 2.6.x kernel, then you might see if the application systemTap will help you to diagnose the problem. You might also check if any cron job is running on this 4-minute cycle.

The other obvious suspects here are device drivers. Does lsmod show anything running that's out of the ordinary? Do you have WiFi running on this system (if som using what deirvers)?

Finally, 50-95% is a pretty big range. It makes me wonder if there is ALSO some process grabbing a lot of CPU (the other 50%, when this value is low) on a regular basis. Do you see anything in top?

If none of this helps, and/or if any of my guesses have been wrong, you might post again with a more complete description of the system, namely: what distro and version, what kernel ("uname -a" usually is enough), and what type and speed of CPU. The output of "free" is sometimes instructive as well. And what is the typical system load (1.00 - the "id" value as reported by top on its third line)?

Oh, and you you use this system for anything other than the usual sorts of desktop and server uses?

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

author
#8 lore raw

another place where bootup messages are recorded?

Recently I have begun to get a great number of messages at boot-up which are about 'querying' scsi drives. I need to get a better look at the messages, but cannot find them in the logs in /var/log. Is there another place where bootup messages are recorded?

Thanks,

Tom Arnall
north spit, ca

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

#9 lore raw

Re: another place where bootup messages are recorded?

Quoted 8 lines · tom arnall

Recently I have begun to get a great number of messages at boot-up which are about 'querying' scsi drives. I need to get a better look at the messages, but cannot find them in the logs in /var/log. Is there another place where bootup messages are recorded?

Thanks,

Tom Arnall
north spit, ca

man dmesg
?
hth, Chuck

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

#10 lore raw

Re: another place where bootup messages are recorded?

Quoted 6 lines · Chuck Gelm
Quoted 4 lines · tom arnall

Recently I have begun to get a great number of messages at boot-up which are about 'querying' scsi drives. I need to get a better look at the messages, but cannot find them in the logs in /var/log. Is there another place where bootup messages are recorded?

Thanks,

Quoted 7 lines
Quoted 4 lines

Tom Arnall
north spit, ca

man dmesg
?
hth, Chuck

In addition to what this man page tells you, see if you have a file called /var/log/dmesg . Many (all?) distros these days dump the contents of the dmesg buffer here during boot/init ... handy because dmesg accesses a ring buffer, and later messages may have overwritten the boot-time ones you want to see.

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

A message that is not in the archive (sent off-list or not yet received): 20060829205333.GC1633@slug

author
#12 lore raw

Re: another place where bootup messages are recorded?

Quoted 10 lines · you
Quoted 4 lines · tom arnall

Recently I have begun to get a great number of messages at boot-up which are about 'querying' scsi drives. I need to get a better look at the messages, but cannot find them in the logs in /var/log. Is there another place where bootup messages are recorded?

You'll have them printed by typing 'dmesg'
Regards,
Frederik

the messages aren't in there either ;( . the weird thing about them is they say that the boot process is checking a scsi device, of which i have none on my system. this phenomenon started btw after i installed the following:

suspend2
software to configure the touchpad
laptop mode
acpi-related stuff

also simultaneous to the appearance of the new messages was the appearance of a recurring 'mini-hang' in the system. the hang lasts about one second and occurs every 4 mins 2 secs exactly, except for a short period after booting, when the intervals are smaller though still in a precise pattern. whenever the hang occurs, the fan comes on for about 10 seconds.

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

author
#13 lore raw
In reply to [email protected]
Quoted 30 lines · On Saturday 26 August 2006 18:22, [email protected] wrote:

On Sat, 26 Aug 2006 12:55:33 -0700

Quoted 3 lines · tom arnall

My system hangs momentarily every 4 mins 2 secs. The event coincides with a 'top' entry in which 'events/0' grabs 50-95% of cpu capacity. How can I determine what is causing the event?

First of all, I'm a newbie. But I'm interested in your subject. What about trying 'pstree'

$ pstree -p
<example output>
 |--events/0(3)--|-aio/0(44)
 |
                 |-kacpid(5)
                 |-kblockd/0(31)
                 |-khelper(4)
                 |-pdflush(41)
                 |-pdflush(42)

<example output>

Then take the pid's and use top to monitor them.

$ top -p3,44,5,31,4,41,42

I don't know if the subtrees will show you anything, but maybe worth a try.

Mike

Signature · 9 lines
thanks,

tom arnall
north spit, ca
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#16 lore raw
Quoted 5 lines · John Kelly

On Wed, 30 Aug 2006 14:35:12 -0700, tom arnall <[email protected]>

Quoted 1 line · wrote:

Is there another place where bootup messages are recorded?

Quoted 8 lines

Does your /etc/default/bootlogd say:

Quoted 2 lines

# Run bootlogd at startup ?
BOOTLOGD_ENABLE=yes

That captures additional messages in /var/log/boot

Be cautious though, if using a serial console. With bootlogd enabled, I had one system lock up at boot, when using a serial console.

thanks for the info. indeed it works: /var/log/boot has all the boot messages.

now for the messages of interest. i have been getting them as of ~1 month ago. their appearance coincided with the appearance of another phenomenon: my system momentarily hangs every 4 mins 2 secs exactly. by 'momentarily hangs' i mean the CPU usage jumps to 50-99% usage (by events/0) and the system doesn't respond to keyboard or mouse input. this lasts for about 1 second. the messages are these, abbreviated somewhat:

Remounting filesystems.
/dev/hda1 not found in PARTITIONS.
/ not found in PARTITIONS.
Checking /dev/hda1 against HD because PARTITIONS contains "auto".
Considering /dev/hda.
/dev/hda1 contains /dev/hda, which is in HD, so we will remount it. Considering /dev/hdb.
o
o
o
Considering /dev/sdg.
Considering /dev/sdh.
Original options: rw,notail
Reducing file system type.
No saved mount options, so apparently we never remounted this filesystem
during this session.
Not remounting.
Executing: blockdev --setra 256 /dev/hda1
proc not found in PARTITIONS.
/proc not found in PARTITIONS.
Checking proc against HD because PARTITIONS contains "auto".
Considering /dev/hda.
Considering /dev/hdb.
o
o
o
Considering /dev/sdf.
Considering /dev/sdg.
Considering /dev/sdh.
sysfs not found in PARTITIONS.
/sys not found in PARTITIONS.
Checking sysfs against HD because PARTITIONS contains "auto".
Considering /dev/hda.
Considering /dev/hdb.
o
o
o
Considering /dev/sdf.
Considering /dev/sdg.
Considering /dev/sdh.
devpts not found in PARTITIONS.
/dev/pts not found in PARTITIONS.
Checking devpts against HD because PARTITIONS contains "auto".
Considering /dev/hda.
Considering /dev/hdb.
o
o
o
Considering /dev/sdh.
tmpfs not found in PARTITIONS.
/dev/shm not found in PARTITIONS.
Checking tmpfs against HD because PARTITIONS contains "auto".
Considering /dev/hda.
Considering /dev/hdb.
o
o
o
Considering /dev/sdg.
Considering /dev/sdh.
usbfs not found in PARTITIONS.
/proc/bus/usb not found in PARTITIONS.
Checking usbfs against HD because PARTITIONS contains "auto".
Considering /dev/hda.
Considering /dev/hdb.
o
o
o
Considering /dev/sdg.
Considering /dev/sdh.
Setting spindown timeout on drives to 7200 seconds.
(hdparm configuration value = 244.)
Querying /dev/hda media type using device name: type 'disk' on bus 'ata'
detected
Querying /dev/hda media type using device name: type 'disk' on bus 'ata'
detected
Executing: hdparm -S 244 /dev/hda
/dev/hda: setting standby to 244 (2 hours)
Querying /dev/hdb media type using device name: failed - unknown name Querying /dev/hdb type using hdparm: type 'disk' on bus 'ata' detected Querying /dev/hdb media type using device name: failed - unknown name Querying /dev/hdb type using hdparm: type 'disk' on bus 'ata' detected
Executing: hdparm -S 244 /dev/hdb
/dev/hdb: No such device or address
Querying /dev/hdc media type using device name: type 'cd' on bus 'ata'
detected
Skipping /dev/hdc: idle timeout control not supported. Querying /dev/hdd media type using device name: failed - unknown name
o
o
o
Querying /dev/hdh media type using device name: failed - unknown name Querying /dev/hdh type using hdparm: type 'disk' on bus 'ata' detected Querying /dev/hdh media type using device name: failed - unknown name Querying /dev/hdh type using hdparm: type 'disk' on bus 'ata' detected
Executing: hdparm -S 244 /dev/hdh
/dev/hdh: No such device or address
Querying /dev/sda media type using device name: type 'disk' on bus 'scsi'
detected
Querying /dev/sda media type using device name: type 'disk' on bus 'scsi'
detected
Executing: hdparm -S 244 /dev/sda
/dev/sda: No such device or address
Querying /dev/sdb media type using device name: type 'disk' on bus 'scsi'
detected
Querying /dev/sdb media type using device name: type 'disk' on bus 'scsi'
detected
Executing: hdparm -S 244 /dev/sdb
/dev/sdb: No such device or address
o
o
o
Querying /dev/sdg media type using device name: type 'disk' on bus 'scsi'
detected
Querying /dev/sdg media type using device name: type 'disk' on bus 'scsi'
detected
Executing: hdparm -S 244 /dev/sdg
/dev/sdg: No such device or address
Querying /dev/sdh media type using device name: type 'disk' on bus 'scsi'
detected
Querying /dev/sdh media type using device name: type 'disk' on bus 'scsi'
detected
Executing: hdparm -S 244 /dev/sdh
/dev/sdh: No such device or address

thanks in advance,

tom arnall
north spit, ca

Signature · 6 lines
VGER BF report: U 0.500196
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs