Skip to contentSkip to conversation

dm-latency: Introduction

PATCH In review Coly Li 4 participants 9 messages Feb 26, 2015 – Feb 27, 2015 Device mapper

lore
Share & tools
lore

Coly Li

author
#0 lore raw

cover dm-latency: Introduction

From: Coly Li <[email protected]>

Dm-latency patch set is an effort to measure hard disk I/O latency on top of device mapper layer. The original motivation of I/O latency measurement was to predict hard disk failure by machine learning method, I/O latency information was one of the inputs sent to machine learning model.

This patch set was written in Aug~Sep 2013, I deployed it on many servers of Alibaba cloud infrastructure. After running for weeks, some interesting data about hard disk I/O latency was observed. In 2013, I gave a talk on OpenSuSE Conference about this topic
(http://blog.coly.li/docs/osc13-coly.pdf).

When generating time stamp for I/O request, clock source is a global unique resource which is protected by spin-locks. Dm-latency was tested on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there will be spin-lock scalability issue, when accessing clock source for time stamping.

Dm-latency is good for I/O latency measurement to hard disk based storage, no matter local or distributed storage via network. For PCI-e or NVMe SSD, I suggest people to look for device provided statistic information, if there is.

The code is very simple, there is no resource allocation/destory, no spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel more than 1 year, no bug reported in last 12 months.

This patch set has 4 patches,
- [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
- [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
- [PATCH 3/4] dm-latency: add sysfs interface
- [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs interface
All these patches are rebased on Linux 4.0-rc1.

Today Laurence Oberman from Redhat sent me an email asking whether this patch set is upstream merged, because he is thinking of pulling this patch set into their kernel. I'd like to maintain this patch set, hope it could be merged.

Thanks in advance.

Coly Li

maintainer
#1 lore raw

Hi

We have already dm-statistics that counts various events - see Documentation/device-mapper/statistics.txt. It counts the nubmer of requests and the time spent servicing each request, thus you can calculate average latency from these values.

Please look at dm-statistics to see if it fits your purpose. If you need additional information not provided by dm-statistics, it would be better to extend the statistics code rather than introduce new "latency" infrastructure.

Mikulas

Quoted 55 lines · Coly Li

From: Coly Li <[email protected]>

Dm-latency patch set is an effort to measure hard disk I/O latency on top of device mapper layer. The original motivation of I/O latency measurement was to predict hard disk failure by machine learning method, I/O latency information was one of the inputs sent to machine learning model.

This patch set was written in Aug~Sep 2013, I deployed it on many servers of Alibaba cloud infrastructure. After running for weeks, some interesting data about hard disk I/O latency was observed. In 2013, I gave a talk on OpenSuSE Conference about this topic
(http://blog.coly.li/docs/osc13-coly.pdf).

When generating time stamp for I/O request, clock source is a global unique resource which is protected by spin-locks. Dm-latency was tested on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there will be spin-lock scalability issue, when accessing clock source for time stamping.

Dm-latency is good for I/O latency measurement to hard disk based storage, no matter local or distributed storage via network. For PCI-e or NVMe SSD, I suggest people to look for device provided statistic information, if there is.

The code is very simple, there is no resource allocation/destory, no spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel more than 1 year, no bug reported in last 12 months.

This patch set has 4 patches,
- [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
- [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
- [PATCH 3/4] dm-latency: add sysfs interface
- [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs interface
All these patches are rebased on Linux 4.0-rc1.

Today Laurence Oberman from Redhat sent me an email asking whether this patch set is upstream merged, because he is thinking of pulling this patch set into their kernel. I'd like to maintain this patch set, hope it could be merged.

Thanks in advance.

Coly Li

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

#2 lore raw

Mikulas
Thanks
This came from customer asking for support for similar functionality in proprietary solutions such as PowerPath and HDLM. I had seen the information from Coly Li and asked if he could submit for comments.
I will look into what can be done with what is is /sys/block/dm-xxx/stat

Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Mikulas Patocka" <[email protected]>
To: "device-mapper development" <[email protected]>, "Coly Li" <[email protected]>
Cc: "Tao Ma" <[email protected]>, "Robin Dong" <[email protected]>, "Laurence Oberman" <[email protected]>, "Alasdair Kergon" <[email protected]>
Sent: Thursday, February 26, 2015 11:49:28 AM
Subject: Re: [dm-devel] [PATCH 0/4] dm-latency: Introduction

Hi

We have already dm-statistics that counts various events - see Documentation/device-mapper/statistics.txt. It counts the nubmer of requests and the time spent servicing each request, thus you can calculate average latency from these values.

Please look at dm-statistics to see if it fits your purpose. If you need additional information not provided by dm-statistics, it would be better to extend the statistics code rather than introduce new "latency" infrastructure.

Mikulas

Quoted 55 lines · Coly Li

From: Coly Li <[email protected]>

Dm-latency patch set is an effort to measure hard disk I/O latency on top of device mapper layer. The original motivation of I/O latency measurement was to predict hard disk failure by machine learning method, I/O latency information was one of the inputs sent to machine learning model.

This patch set was written in Aug~Sep 2013, I deployed it on many servers of Alibaba cloud infrastructure. After running for weeks, some interesting data about hard disk I/O latency was observed. In 2013, I gave a talk on OpenSuSE Conference about this topic
(http://blog.coly.li/docs/osc13-coly.pdf).

When generating time stamp for I/O request, clock source is a global unique resource which is protected by spin-locks. Dm-latency was tested on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there will be spin-lock scalability issue, when accessing clock source for time stamping.

Dm-latency is good for I/O latency measurement to hard disk based storage, no matter local or distributed storage via network. For PCI-e or NVMe SSD, I suggest people to look for device provided statistic information, if there is.

The code is very simple, there is no resource allocation/destory, no spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel more than 1 year, no bug reported in last 12 months.

This patch set has 4 patches,
- [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
- [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
- [PATCH 3/4] dm-latency: add sysfs interface
- [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs interface
All these patches are rebased on Linux 4.0-rc1.

Today Laurence Oberman from Redhat sent me an email asking whether this patch set is upstream merged, because he is thinking of pulling this patch set into their kernel. I'd like to maintain this patch set, hope it could be merged.

Thanks in advance.

Coly Li

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

#3 lore raw
Quoted 5 lines · Laurence Oberman

Mikulas
Thanks
This came from customer asking for support for similar functionality in proprietary solutions such as PowerPath and HDLM. I had seen the information from Coly Li and asked if he could submit for comments.
I will look into what can be done with what is is /sys/block/dm-xxx/stat

Those are just the basic diskstats counters (similar to /proc/diskstats).

You want to look at the stuff described in Documentation/device-mapper/dm-statistics.txt

Regards,
Bryn.

#4 lore raw
In reply to Mikulas Patocka
Quoted 4 lines · Mikulas Patocka

We have already dm-statistics that counts various events - see Documentation/device-mapper/statistics.txt. It counts the nubmer of requests and the time spent servicing each request, thus you can calculate average latency from these values.

Right: average service time (as reported by iostat etc.) is easily derived from the existing stats.

Does the separate latency accounting buy anything additional?

Quoted 4 lines

Please look at dm-statistics to see if it fits your purpose. If you need additional information not provided by dm-statistics, it would be better to extend the statistics code rather than introduce new "latency" infrastructure.

Agreed; I'm working on userspace support for dm-statistics at the moment and if there is a need for these additional measurements I would greatly prefer to consume them as additional fields in the existing dm-stats counter set.

This also has the advantage of benefiting from the existing step and area support allowing a device to be subdivided into discrete stats regions.

Regards,
Bryn.

maintainer
#5 lore raw
Quoted 21 lines · Bryn M. Reeves
Quoted 4 lines · Mikulas Patocka

We have already dm-statistics that counts various events - see Documentation/device-mapper/statistics.txt. It counts the nubmer of requests and the time spent servicing each request, thus you can calculate average latency from these values.

Right: average service time (as reported by iostat etc.) is easily derived from the existing stats.

Does the separate latency accounting buy anything additional?

Quoted 4 lines

Please look at dm-statistics to see if it fits your purpose. If you need additional information not provided by dm-statistics, it would be better to extend the statistics code rather than introduce new "latency" infrastructure.

Agreed; I'm working on userspace support for dm-statistics at the moment and if there is a need for these additional measurements I would greatly prefer to consume them as additional fields in the existing dm-stats counter set.

Quoted 6 lines

This also has the advantage of benefiting from the existing step and area support allowing a device to be subdivided into discrete stats regions.

Regards,
Bryn.

Coly's paper (http://blog.coly.li/docs/osc13-coly.pdf) shows that they take histogram of latencies and use it to predict disk failure.

That could be easily added to dm-statistics.

Average latency alone can't be used to predict disk failure because average latency depends on the type of workload (for example - sequantial or nearly sequential requests have much lower latency than random requests).

I'd like to know if we need separate histogram per region, or if it is sufficient to have a histogram per device. dm-latency has no regions, it has a histogram for the whole device. The histogram-per-region would consume more memory, I'm interested if there is some reasonable use case for that.

Mikulas

#6 lore raw

For my particular use case its about providing the ability to warn on latencies being seen for multipath devices based on a given threshold. Of course this can simply be a userspace tool using what we already expose and do the calculations to make it work. When we have these latencies we then focus on which SAN path may or may not be contributing.
Within multipathd we can already configure service time as a load balancer, perhaps we can do the monitoring in the same place. i.e. warn on service time above a threshold.

When a customer says "I currently use the following xxxx for multipath on RHEL however I want to go to native multipathing, but you don't provide the monitoring I need" I want to work to an enhancement.

For example I had Ben add the ability just recently to multipath to control re-establishing path usage based on path health when the path returns to mimic what DMP can do to avoid recovery due to path flapping.

Thanks

Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Mikulas Patocka" <[email protected]>
To: "Bryn M. Reeves" <[email protected]>
Cc: "device-mapper development" <[email protected]>, "Tao Ma" <[email protected]>, "Robin Dong" <[email protected]>, "Laurence Oberman" <[email protected]>, "Coly Li" <[email protected]>, "Alasdair Kergon" <[email protected]>
Sent: Thursday, February 26, 2015 2:34:40 PM
Subject: Re: [dm-devel] [PATCH 0/4] dm-latency: Introduction

Quoted 21 lines · Bryn M. Reeves
Quoted 4 lines · Mikulas Patocka

We have already dm-statistics that counts various events - see Documentation/device-mapper/statistics.txt. It counts the nubmer of requests and the time spent servicing each request, thus you can calculate average latency from these values.

Right: average service time (as reported by iostat etc.) is easily derived from the existing stats.

Does the separate latency accounting buy anything additional?

Quoted 4 lines

Please look at dm-statistics to see if it fits your purpose. If you need additional information not provided by dm-statistics, it would be better to extend the statistics code rather than introduce new "latency" infrastructure.

Agreed; I'm working on userspace support for dm-statistics at the moment and if there is a need for these additional measurements I would greatly prefer to consume them as additional fields in the existing dm-stats counter set.

Quoted 6 lines

This also has the advantage of benefiting from the existing step and area support allowing a device to be subdivided into discrete stats regions.

Regards,
Bryn.

Coly's paper (http://blog.coly.li/docs/osc13-coly.pdf) shows that they take histogram of latencies and use it to predict disk failure.

That could be easily added to dm-statistics.

Average latency alone can't be used to predict disk failure because average latency depends on the type of workload (for example - sequantial or nearly sequential requests have much lower latency than random requests).

I'd like to know if we need separate histogram per region, or if it is sufficient to have a histogram per device. dm-latency has no regions, it has a histogram for the whole device. The histogram-per-region would consume more memory, I'm interested if there is some reasonable use case for that.

Mikulas

#7 lore raw
Quoted 5 lines · Laurence Oberman

For my particular use case its about providing the ability to warn on latencies being seen for multipath devices based on a given threshold. Of course this can simply be a userspace tool using what we already expose and do the calculations to make it work. When we have these latencies we then focus on which SAN path may or may not be contributing.
Within multipathd we can already configure service time as a load balancer, perhaps we can do the monitoring in the same place. i.e. warn on service time above a threshold.

One limitation here is that the dm-mpath target is request-based. Currently dm-statistics are only available for bio-based targets. This means that to obtain fine-grained stats for multipath devices we need to insert a linear layer on top of the dm-mpath device.

If we wanted to take this further and apply counters to each underlying path then that would require a similar layer to be added between the sd devices and the dm-mpath map.

Quoted 3 lines

When a customer says "I currently use the following xxxx for multipath on RHEL however I want to go to native multipathing, but you don't provide the monitoring I need" I want to work to an enhancement.

Average wait times are easily obtained from the current kernel counter set but (aside from the ability to hone in on subsections of the device) this doesn't buy you that much beyond vanilla iostat.

Regards,
Bryn.

maintainer
#8 lore raw
Quoted 11 lines · Bryn M. Reeves
Quoted 5 lines · Laurence Oberman

For my particular use case its about providing the ability to warn on latencies being seen for multipath devices based on a given threshold. Of course this can simply be a userspace tool using what we already expose and do the calculations to make it work. When we have these latencies we then focus on which SAN path may or may not be contributing.
Within multipathd we can already configure service time as a load balancer, perhaps we can do the monitoring in the same place. i.e. warn on service time above a threshold.

One limitation here is that the dm-mpath target is request-based. Currently dm-statistics are only available for bio-based targets. This means that to obtain fine-grained stats for multipath devices we need to insert a linear layer on top of the dm-mpath device.

Someone could hack dm statistics to work on request based targets.

Mikulas