Skip to contentSkip to conversation

Why is deleting (or reading) files not counted as IO-Wait in top?

DISCUSSION Matthias Schniedermeyer 4 participants 6 messages Jan 2, 2008 – Jan 14, 2008

lore
Share & tools
lore
#0 lore raw

Hi

Currently i'm deleting about 500.000 files on a XFS-filesystem which takes a few minutes, as i had a top open i saw that 'wa' is shown as 0.0% (Nothing else running currently) and everything except 'id' is near the bottom too. Kernel is 2.6.23.11.

So, as 'rm -rf' is essentially a IO (or seek, to be more correct)-bound task, shouldn't that count as "Waiting for IO"?

The man-page of top says:
'Amount of time the CPU has been waiting for I/O to complete.'

But AFAICT wa only seams to be (ac)counted for writing and not for reading. I come to that conclusion because, when i fire 'sync' i can see some percent wa for a few seconds.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#1 lore raw
Quoted 14 lines · On Wednesday, 2 January 2008 21:35:03 Matthias Schniedermeyer wrote:

Hi

Currently i'm deleting about 500.000 files on a XFS-filesystem which takes a few minutes, as i had a top open i saw that 'wa' is shown as 0.0% (Nothing else running currently) and everything except 'id' is near the bottom too. Kernel is 2.6.23.11.

So, as 'rm -rf' is essentially a IO (or seek, to be more correct)-bound task, shouldn't that count as "Waiting for IO"?

The man-page of top says:
'Amount of time the CPU has been waiting for I/O to complete.'

Quoted 9 lines

But AFAICT wa only seams to be (ac)counted for writing and not for reading. I come to that conclusion because, when i fire 'sync' i can see some percent wa for a few seconds.

Bis denn

The IOWAIT time is the IDLE time that was spent waiting
for I/O. (meaning that there were no tasks running, but some were waiting on I/O)

Thus if you have another task that is not I/O bound, it can run in that time, and ideally, you shouldn't notice any I/O slowdown, but the iowait time will decrease.

It wasn't the case before CFS introduction. I did few tests that showed almost 50% slowdown when running another task in that iowait time.
It is not longer a problem with CFS.

Regards,
Maxim Levitsky

#2 lore raw
Quoted 22 lines · Maxim Levitsky
Quoted 11 lines · On Wednesday, 2 January 2008 21:35:03 Matthias Schniedermeyer wrote:

Hi

Currently i'm deleting about 500.000 files on a XFS-filesystem which takes a few minutes, as i had a top open i saw that 'wa' is shown as 0.0% (Nothing else running currently) and everything except 'id' is near the bottom too. Kernel is 2.6.23.11.

So, as 'rm -rf' is essentially a IO (or seek, to be more correct)-bound task, shouldn't that count as "Waiting for IO"?

Quoted 6 lines

The man-page of top says:
'Amount of time the CPU has been waiting for I/O to complete.'

But AFAICT wa only seams to be (ac)counted for writing and not for reading. I come to that conclusion because, when i fire 'sync' i can see some percent wa for a few seconds.

The IOWAIT time is the IDLE time that was spent waiting
for I/O. (meaning that there were no tasks running, but some were waiting on I/O)

Quoted 6 lines

Thus if you have another task that is not I/O bound, it can run in that time, and ideally, you shouldn't notice any I/O slowdown, but the iowait time will decrease.

It wasn't the case before CFS introduction. I did few tests that showed almost 50% slowdown when running another task in that iowait time.
It is not longer a problem with CFS.

I can understand that, but in my case nothing else was running, so i would expect about 46%-50% wa (Dual Core Processor).

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#3 lore raw
In reply to Matthias Schniedermeyer
Quoted 11 lines · On Wed, 2 Jan 2008 20:35:03 +0100 Matthias Schniedermeyer <[email protected]> wrote:

Hi

Currently i'm deleting about 500.000 files on a XFS-filesystem which takes a few minutes, as i had a top open i saw that 'wa' is shown as 0.0% (Nothing else running currently) and everything except 'id' is near the bottom too. Kernel is 2.6.23.11.

So, as 'rm -rf' is essentially a IO (or seek, to be more correct)-bound task, shouldn't that count as "Waiting for IO"?

Quoted 7 lines

The man-page of top says:
'Amount of time the CPU has been waiting for I/O to complete.'

But AFAICT wa only seams to be (ac)counted for writing and not for reading. I come to that conclusion because, when i fire 'sync' i can see some percent wa for a few seconds.

Yes, you would absolutely expect `rm' to be stuck in D state and contributing to both load average and io-wait in this situation.

I'd think that either XFS is playing games (and it'd take some pretty inventive games to do this) or your observations are in error.

#4 lore raw
Quoted 21 lines · Andrew Morton
Quoted 11 lines · On Wed, 2 Jan 2008 20:35:03 +0100 Matthias Schniedermeyer <[email protected]> wrote:

Hi

Currently i'm deleting about 500.000 files on a XFS-filesystem which takes a few minutes, as i had a top open i saw that 'wa' is shown as 0.0% (Nothing else running currently) and everything except 'id' is near the bottom too. Kernel is 2.6.23.11.

So, as 'rm -rf' is essentially a IO (or seek, to be more correct)-bound task, shouldn't that count as "Waiting for IO"?

Quoted 7 lines

The man-page of top says:
'Amount of time the CPU has been waiting for I/O to complete.'

But AFAICT wa only seams to be (ac)counted for writing and not for reading. I come to that conclusion because, when i fire 'sync' i can see some percent wa for a few seconds.

Quoted 5 lines

Yes, you would absolutely expect `rm' to be stuck in D state and contributing to both load average and io-wait in this situation.

I'd think that either XFS is playing games (and it'd take some pretty inventive games to do this) or your observations are in error.

I don't think that i could do much wrong(tm).

rm -rf <big dir> (Took >30 minutes to delete)
Everything else was sleeping at the time.

The sleeping processes are the usual handful of about 200 processes, you have laying around today. 47 kernel threads, another about 40 or so demons, and the rest >100 processes was my KDE-Session (mostly 'konsole's with bashes inside) with Iceweasel (Firefox) as sugar on top. But as i said, there weren't doing anything at all, 'us' peaked at about 1-1.4 when i watched top.

Bis denn

Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.
#5 lore raw
In reply to Matthias Schniedermeyer
Quoted 7 lines · Matthias Schniedermeyer

Hi

Currently i'm deleting about 500.000 files on a XFS-filesystem which takes a few minutes, as i had a top open i saw that 'wa' is shown as 0.0% (Nothing else running currently) and everything except 'id' is near the bottom too. Kernel is 2.6.23.11.

Simply because the only I/O that XFS does during a delete is to the log and the log does async I/O and hence the process never blocks in I/O.

Instead, it blocks in a far more complex space reservation that may or may not be related to I/O wait....

Quoted 2 lines

So, as 'rm -rf' is essentially a IO (or seek, to be more correct)-bound task, shouldn't that count as "Waiting for IO"?

rm -rf is not seek bound on XFS - it's generally determined by the sequential write speed of the block device or how fast your CPU is....

Quoted 2 lines

The man-page of top says:
'Amount of time the CPU has been waiting for I/O to complete.'

Async I/O means that typically your CPU does not get held up waiting for I/O to complete....

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group