Skip to contentSkip to conversation

2.6.21.1 - 97% wait time on IDE operations

DISCUSSION Tommy Vercetti 9 participants 15 messages May 26, 2007 – May 29, 2007

lore
Share & tools
lore
#0 lore raw

Hi folks,

I was trying to get answer to my question around, but no one knows. I do have DMA turned on, etc, yet - on extensive harddrive operations wait time is 90+% , which means that machine is waiting, rather than doing something meanwhile. (I guess).
Can someone describe to me , in more detail why is that happening, and what steps should I consider to avoid it ? I couldn't find any answers that would have help me on net.

thanks.

Signature · 1 line
Vercetti
#1 lore raw
Quoted 7 lines · On Sat, 26 May 2007 15:05:38 +0200 Tommy Vercetti <[email protected]> wrote:

I was trying to get answer to my question around, but no one knows. I do have DMA turned on, etc, yet - on extensive harddrive operations wait time is 90+% , which means that machine is waiting, rather than doing something meanwhile. (I guess).
Can someone describe to me , in more detail why is that happening, and what steps should I consider to avoid it ? I couldn't find any answers that would have help me on net.

It means that the disk is slow and the CPU is fast... so while the disk is busy seeking and reading data the CPU has nothing to do but wait for it.

Idle == CPU has nothing to do
Waiting == CPU has nothing to do, but it will have as soon as the slow
disk (or whatever) delivers data

Anyway 97% is quite high... what CPU / Hard Disk do you have?

What kernel version?
I/O scheduler? (cat /sys/block/DEVICE/queue/scheduler) Filesystem?

And what time of "operations" are you doing?

Signature · 2 lines
	Paolo Ornati
	Linux 2.6.22-rc3-cfs-v14-gf193016a on x86_64
#2 lore raw
Quoted 8 lines · Paolo Ornati

On Sat, 26 May 2007 15:05:38 +0200
It means that the disk is slow and the CPU is fast... so while the disk is busy seeking and reading data the CPU has nothing to do but wait for it.

Idle == CPU has nothing to do
Waiting == CPU has nothing to do, but it will have as soon as the slow
disk (or whatever) delivers data

mhm.

Quoted 1 line

Anyway 97% is quite high... what CPU / Hard Disk do you have?

Intel(R) Pentium(R) M processor 1400MHz
TOSHIBA MK8025GAS

Quoted 1 line

What kernel version?

2.6.21.1

Quoted 1 line

I/O scheduler? (cat /sys/block/DEVICE/queue/scheduler)

gj@puppet:~$ cat /sys/block/hda/queue/scheduler
noop anticipatory deadline [cfq]
Quoted 1 line

Filesystem?

reiser3

Quoted 1 line

And what time of "operations" are you doing?

apt-get install, vmware

Signature · 1 line
Vercetti
#3 lore raw
Quoted 6 lines · On Sat, 26 May 2007 16:07:49 +0200 Tommy Vercetti <[email protected]> wrote:
Quoted 1 line

Anyway 97% is quite high... what CPU / Hard Disk do you have?

Intel(R) Pentium(R) M processor 1400MHz
TOSHIBA MK8025GAS

Quoted 1 line

What kernel version?

2.6.21.1

Quoted 8 lines
Quoted 1 line

I/O scheduler? (cat /sys/block/DEVICE/queue/scheduler)

gj@puppet:~$ cat /sys/block/hda/queue/scheduler
noop anticipatory deadline [cfq]
Quoted 1 line

Filesystem?

reiser3

Quoted 1 line

And what time of "operations" are you doing?

apt-get install, vmware

It's probably just a slow disk... try hdparm as suggested by Ray and see if they look sane (PS: just seen the numbers and looks sane to me).

If they are and you need more performance buy a better HD :)

Improvements from kernel side are possible but don't expect too much.

Things to check/try:

1) FS: is it in good shape? Or is it full and fragmented?

2) FS mount options, try adding everything that can reduce accesses (noatime, nodiratime)

3) try experimental patches such as Adaptive Readahead, it seems to help some workload (it was there in older -mm, now there's a much simplier readahead replacement in 2.6.22-rc2-mm1 called "on-demand readahead", maybe that helps too?)

Signature · 2 lines
	Paolo Ornati
	Linux 2.6.22-rc3-cfs-v14-gf193016a on x86_64
#4 lore raw
In reply to Tommy Vercetti
Quoted 9 lines · Tommy Vercetti

Hi folks,

I was trying to get answer to my question around, but no one knows. I do have DMA turned on, etc, yet - on extensive harddrive operations wait time is 90+% , which means that machine is waiting, rather than doing something meanwhile. (I guess).
Can someone describe to me , in more detail why is that happening, and what steps should I consider to avoid it ? I couldn't find any answers that would have help me on net.

I see in your other post that you have a laptop disk drive and, based on its specs, it is a 4200RPM one. Laptop drives are incredibly slow for random disk access -- I have one and hate it with passion... even though it is slightly better (5400RPM)...

You mention that you are doing expensive disk operations. If these involve a lot of disk seeks (such as, for example, a "cvs update" on a large source tree), disk performance will be very very poor and any other operation you attempt on it will be slow as well. These kind of drives are the major bottleneck of current laptop machines, in my opinion.

(I just wanted to make this clear wrt the hardware. I do not know if Linux is also involved or not.)

Signature · 1 line
Julio M. Merino Vidal <[email protected]>
#5 lore raw
In reply to Tommy Vercetti
Quoted 2 lines · Tommy Vercetti

I was trying to get answer to my question around, but no one knows. I do have DMA turned on, etc, yet - on extensive harddrive operations wait

You may have DMA turned on, but it may be ineffectual if you don't have the right IDE driver handing your disk.

Quoted 2 lines

time is 90+% , which means that machine is waiting, rather than doing something meanwhile. (I guess).

What does hdparm -tT /dev/hda say? If the numbers are reasonable (~400MB/s for cache reads, ~25MB/s for buffered disk reads), then you're just doing a lot of seeking on the drive, and that takes a long time. (Drives can seek about 100 times a second. If every seek reads only 4k of data, that's 400k of data processed per second, or a lot of time spent with the CPU waiting on I/O.)

#6 lore raw

in terms of driver, what do you mean ?
this is my lspci:
00:00.0 Host bridge: Intel Corporation 82855PM Processor to I/O Controller (rev 21)
00:01.0 PCI bridge: Intel Corporation 82855PM Processor to AGP Controller (rev
21)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 83) 00:1f.0 ISA bridge: Intel Corporation 82801DBM (ICH4-M) LPC Interface Bridge (rev 03)
00:1f.1 IDE interface: Intel Corporation 82801DBM (ICH4-M) IDE Controller (rev
03)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 03)
00:1f.6 Modem: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Modem Controller (rev 03)
01:00.0 VGA compatible controller: nVidia Corporation NV34M [GeForce FX Go5200] (rev a1)
02:07.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (MOB) Ethernet Controller (rev 83)
02:0a.0 Network controller: Intel Corporation PRO/Wireless LAN 2100 3B Mini PCI Adapter (rev 04)
02:0b.0 CardBus bridge: Toshiba America Info Systems ToPIC100 PCI to Cardbus Bridge with ZV Support (rev 33)
02:0d.0 System peripheral: Toshiba America Info Systems SD TypA Controller (rev 05)
03:00.0 Ethernet controller: Atheros Communications, Inc. AR5212 802.11abg NIC (rev 01)

modules list, and config on request (don't want to pollute devs mailboxes).

/dev/hda:
Timing cached reads: 732 MB in 2.00 seconds = 365.74 MB/sec
Timing buffered disk reads: 76 MB in 3.00 seconds = 25.31 MB/sec

root@puppet:~# htparm -i -v /dev/hda
-su: htparm: command not found
root@puppet:~# hdparm -i -v /dev/hda

/dev/hda:

 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 65535/16/63, sectors = 156301488, start = 0

Model=TOSHIBA MK8025GAS, FwRev=KA023A, SerialNo=95NX6153S Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=48 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=16 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=156301488 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: sdma0 sdma1 sdma2 mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma3 udma4 *udma5 AdvancedPM=yes: unknown setting WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3
ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6

* signifies the current active mode

Signature · 1 line
Vercetti
#7 lore raw
Quoted 1 line · Tommy Vercetti

in terms of driver, what do you mean ?

The software driver that handles the specific chipset for that drive. The thing that shows up when you do an lsmod, assuming you have the driver compiled as a module.

Quoted 3 lines

/dev/hda:
Timing cached reads: 732 MB in 2.00 seconds = 365.74 MB/sec
Timing buffered disk reads: 76 MB in 3.00 seconds = 25.31 MB/sec

That looks fine. Your filesystem may just be very fragmented. I'm not familiar with how reiser3 filesystems age, so if you've had the system installed for quite some time, it may be time to back it up and reinstall. Dunno.

All of Paolo's suggestions look good, BTW. Sorry I couldn't be more help.

#8 lore raw
Quoted 6 lines · Ray Lee
Quoted 1 line · Tommy Vercetti

in terms of driver, what do you mean ?

The software driver that handles the specific chipset for that drive. The thing that shows up when you do an lsmod, assuming you have the driver compiled as a module.

root@puppet:~# dmesg |grep -i ide
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1
    ide0: BM-DMA at 0x1840-0x1847, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x1848-0x184f, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
ide1 at 0x170-0x177,0x376 on irq 15

it's compiled in.
you reckon I should use something else for this platform ?

Quoted 8 lines
Quoted 3 lines

/dev/hda:
Timing cached reads: 732 MB in 2.00 seconds = 365.74 MB/sec
Timing buffered disk reads: 76 MB in 3.00 seconds = 25.31 MB/sec

That looks fine. Your filesystem may just be very fragmented. I'm not familiar with how reiser3 filesystems age, so if you've had the system installed for quite some time, it may be time to back it up and reinstall. Dunno.

yeah, and they say unix FSes don't need defragmentation.... my bottom...

Signature · 1 line
Vercetti
#9 lore raw
Quoted 5 lines
Quoted 4 lines

That looks fine. Your filesystem may just be very fragmented. I'm not familiar with how reiser3 filesystems age, so if you've had the system installed for quite some time, it may be time to back it up and reinstall. Dunno.

yeah, and they say unix FSes don't need defragmentation.... my bottom...

Provided they don't get within about 5-10% of full a lot of the time Unix file systems generally don't - especially with preallocators. Reiser3 is not a traditional unix file system, its very different to almost any other fs, and it does need defragmenting.

#10 lore raw

Hi, Alan!

Quoted 6 lines · Alan Cox
Quoted 1 line

yeah, and they say unix FSes don't need defragmentation.... my bottom...

Provided they don't get within about 5-10% of full a lot of the time Unix file systems generally don't - especially with preallocators. Reiser3 is not a traditional unix file system, its very different to almost any other fs, and it does need defragmenting.

Just FYI:
http://ck.kolivas.org/apps/defrag/
http://sourceforge.net/projects/davtools/
defragmentation seems to be a minor issue.

After reiser3 (kernel 2.6.18.1 or so) crashed(*) badly with some data loss on a production server can you recommed a _safe_ and good performing filesystem for an compile and test server (lots of small files).

(*) rm ~/Maildir while it was fed with courier-imap and pressed CTRL-C to interrupt that trashed some of the files in ~/Maildir. It needed a --rebuild-tree.

Best greets,

Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm-technology.com
Phone: +49-89-741518-50
Fax: +49-89-741518-19

#11 lore raw
In reply to Alan Cox
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"AC" == Alan Cox <[email protected]> writes:

AC> Provided they don't get within about 5-10% of full a lot of the AC> time Unix file systems generally don't

That's a big if right there. For servers it isn't a problem, few people can get capacity right to withing 10%, so you never let a server run full. Desktops/laptops on the other hand spend most of their lives between 80% and 100%.

/Benny

#12 lore raw

O> That's a big if right there. For servers it isn't a problem, few

Quoted 3 lines

people can get capacity right to withing 10%, so you never let a server run full. Desktops/laptops on the other hand spend most of their lives between 80% and 100%.

Modern desktop patterns are very different to older ones - the disk fill is almost entirely continuous writes of large files (OGG, MP3, Movies etc). In addition the default ext3 behaviour reserves the last part of the disk for root - so usually it doesn't get below 5% free as only root can steal that space.

Back when you had a 40MB /home the usage tended to be multiple parallel writers, permanently at 90%+ (with a "please remove unused files" motd).

Alan

#13 lore raw
In reply to Tommy Vercetti

Hi,

Quoted 2 lines

wait time is 90+% , which means that machine is waiting, rather than doing something meanwhile. (I guess).

Yes, the machine spends its time waiting for the disk to do its things as it does not have anything else to do. Nothing to worry about. If you want, launch some CPU bound process at the same time and the wait time will be replaced by some user+system time but you'll gain nothing.

Hope this helps.

Signature · 1 line
Guillaume
#14 lore raw
In reply to Tommy Vercetti
Quoted 6 lines · Tommy Vercetti

Hi folks,

I was trying to get answer to my question around, but no one knows.
I do have DMA turned on, etc, yet - on extensive harddrive operations wait time is 90+% , which means that machine is waiting, rather than doing something meanwhile. (I guess). Can someone describe to me , in more detail why is that happening, and what steps should I consider to avoid it ? I couldn't find any answers that would have help me on net.

thanks.

From later posts I suspect that your disk performance just sucks, but do use one of the monitoring tools and follow the actual disk work, in term of seeks and transfer rate.

I've been looking at high iowait while disk and network are (nearly) idle, but it sounds as if you just have a bad match of CPU and disk speed.

Can you borrow a USB drive to use for some testing? USB 2 needed to be faster than your old 4200 rpm drive.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot