Skip to contentSkip to conversation

difference between "lat" and "clat"

DISCUSSION Martin Steigerwald 2 participants 3 messages May 9, 2012 – May 10, 2012

lore
Share & tools
lore
#0 lore raw

Hi Jens, hi everyone,

Well mails from my mail client seem to appear here.

I have some questions regarding fio latencies.

There are three kind of latencies described in README, from which fio 2.0.7 only displays clat and lat:

slat = submission latency

clat = completion latency

lat = I/O completion latency

Now I try to understand the README but I am not sure whether I really got the difference.

I understand:

slat: The time to submit the I/O. This is the time after which a syscall returns to userspace. On sync I/O this will include the completion of the request as sync I/O will wait for the request to land in pagecache. On asynchronous I/O this will be really fast.

clat: This is the time to complete handling the I/O request. Then it is at least in the pagecache on buffered I/O.

lat: The time it takes till the request has been processed. The HOWTO says:
"This is the time from when IO leaves fio and when it gets completed."

So how is that completion different from the other completion in clat?

In fact values seem to be roughly the same:

Reads:

    clat (usec): min=773 , max=104267 , avg=6333.91, stdev=4327.57
     lat (usec): min=773 , max=104267 , avg=6334.10, stdev=4327.57

Writes:

    clat (usec): min=773 , max=104267 , avg=6333.91, stdev=4327.57
     lat (usec): min=773 , max=104267 , avg=6334.10, stdev=4327.57

For:

root@frisbie:~#  cat randreadwrite.job
[global]
size=2g
runtime=60
direct=1
[zufälliglesen]
rw=randread
[zufälligschreiben]
rw=randwrite

But even on buffered I/O this seems to be similar (same job without direct=1):

Reads:

    clat (usec): min=189 , max=954818 , avg=5423.72, stdev=21529.85
     lat (usec): min=190 , max=954818 , avg=5423.90, stdev=21529.85

Writes:

    clat (usec): min=3 , max=2391.5K, avg=180.47, stdev=8936.82
     lat (usec): min=4 , max=2391.5K, avg=180.59, stdev=8936.82

So whats the difference here? Is

lat = slat + clat?

Why did you drop slat from output?

I tried to enable it.

But I get:

root@frisbie:~# fio randreadwrite-buffered.job
fio: failed parsing disable_slat=false
fio: job global dropped
root@frisbie:~#

Is this another deprecated manpage option?

And then I try to understand the clat new output:

    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    6],
     | 70.00th=[    6], 80.00th=[    6], 90.00th=[    6], 95.00th=[    7],
     | 99.00th=[   12], 99.50th=[13120], 99.90th=[31360], 99.95th=[38656],
     | 99.99th=[58112]

What does the th mean?

After reading documentation on "percentile_list" read this as follows:

- 1% of the requests have taken 4 usec
- 5% of the requests have taken 5 usec (does that include the 1% number? Seems so)
- 10% of the requests have taken 5 usec (some issue with accuracy here? 5.something usec?)
[…]
- 70% of the requests have taken 6 usec
- 99,99% of the requests have taken less than 58112 usec

Up to know I have looked at the most was "lat" - seems that seemed to be the most realistic value to me. Is there any point in giving much attention to clat from an admin point of view? I am inclined to omit it from output via "disable_clat" unless I see a case where I need this.

Thanks,

Signature · 2 lines
Martin Steigerwald - teamix GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
#1 lore raw

Am Mittwoch, 9. Mai 2012 schrieb Martin Steigerwald:

Quoted 34 lines

Hi Jens, hi everyone,

Well mails from my mail client seem to appear here.

I have some questions regarding fio latencies.

There are three kind of latencies described in README, from which fio 2.0.7 only displays clat and lat:

slat = submission latency

clat = completion latency

lat = I/O completion latency

Now I try to understand the README but I am not sure whether I really got the difference.

I understand:

slat: The time to submit the I/O. This is the time after which a syscall returns to userspace. On sync I/O this will include the completion of the request as sync I/O will wait for the request to land in pagecache. On asynchronous I/O this will be really fast.

clat: This is the time to complete handling the I/O request. Then it is at least in the pagecache on buffered I/O.

lat: The time it takes till the request has been processed. The HOWTO says:
"This is the time from when IO leaves fio and when it gets completed."

So how is that completion different from the other completion in clat?

[…]

Quoted 5 lines

Reads:

    clat (usec): min=189 , max=954818 , avg=5423.72, stdev=21529.85
     lat (usec): min=190 , max=954818 , avg=5423.90, stdev=21529.85
Quoted 8 lines

Writes:

    clat (usec): min=3 , max=2391.5K, avg=180.47, stdev=8936.82
     lat (usec): min=4 , max=2391.5K, avg=180.59, stdev=8936.82

So whats the difference here? Is

lat = slat + clat?

Could be (from another job with iodepth 64)

write:
slat (usec): min=5 , max=753280 , avg=190.49, stdev=8787.41 clat (usec): min=677 , max=2178.2K, avg=317831.81, stdev=282569.45 lat (usec): min=715 , max=2178.2K, avg=318023.20, stdev=282631.85

read:
slat (usec): min=5 , max=139 , avg=22.73, stdev= 5.52 clat (msec): min=8 , max=1756 , avg=203.10, stdev=174.58 lat (msec): min=8 , max=1756 , avg=203.13, stdev=174.58

Hmmm, on read averages don't add up.

Quoted 1 line

Why did you drop slat from output?

Forget this one. It obviously is there.

Ciao,

Signature · 2 lines
Martin Steigerwald - teamix GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
#2 lore raw
Quoted 28 lines · Martin Steigerwald

Am Mittwoch, 9. Mai 2012 schrieb Martin Steigerwald:

Quoted 27 lines

Hi Jens, hi everyone,

Well mails from my mail client seem to appear here.

I have some questions regarding fio latencies.

There are three kind of latencies described in README, from which fio 2.0.7 only displays clat and lat:

slat = submission latency

clat = completion latency

lat = I/O completion latency

Now I try to understand the README but I am not sure whether I really got the difference.

I understand:

slat: The time to submit the I/O. This is the time after which a syscall returns to userspace. On sync I/O this will include the completion of the request as sync I/O will wait for the request to land in pagecache. On asynchronous I/O this will be really fast.

Quoted 21 lines
Quoted 7 lines

clat: This is the time to complete handling the I/O request. Then it is at least in the pagecache on buffered I/O.

lat: The time it takes till the request has been processed. The HOWTO says:
"This is the time from when IO leaves fio and when it gets completed."

So how is that completion different from the other completion in clat?

[…]

Quoted 13 lines

Reads:

    clat (usec): min=189 , max=954818 , avg=5423.72, stdev=21529.85
     lat (usec): min=190 , max=954818 , avg=5423.90, stdev=21529.85

Writes:

    clat (usec): min=3 , max=2391.5K, avg=180.47, stdev=8936.82
     lat (usec): min=4 , max=2391.5K, avg=180.59, stdev=8936.82

So whats the difference here? Is

lat = slat + clat?

Yes, lat is the total latency. slat is the time it takes to submit the IO to the kernel, clat is the time from when slat is over and until the device has completed it (and the application is notified). Hence lat is slat + clat.

Quoted 7 lines

Could be (from another job with iodepth 64)

write:
slat (usec): min=5 , max=753280 , avg=190.49, stdev=8787.41 clat (usec): min=677 , max=2178.2K, avg=317831.81, stdev=282569.45 lat (usec): min=715 , max=2178.2K, avg=318023.20, stdev=282631.85

Quoted 6 lines

read:
slat (usec): min=5 , max=139 , avg=22.73, stdev= 5.52 clat (msec): min=8 , max=1756 , avg=203.10, stdev=174.58 lat (msec): min=8 , max=1756 , avg=203.13, stdev=174.58

Hmmm, on read averages don't add up.

How so? slat is 22.73usec, or 0.002273msec. 203.10 + 0.0023 is roughly 203.13.

Signature · 1 line
Jens Axboe