Backport request to stable of two performance related fixes for xen-blkfront (3.13 fixes to earlier trees)

Quoted 6 lines · Konrad Rzeszutek Wilk

The following distros are affected:
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA

josh

Greg KH

maintainer

2014-05-20 03:19 UTC #3 lore raw

Quoted 31 lines · Konrad Rzeszutek Wilk

Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
I've copied Jerry - the maintainer of the Oracle's kernel. I don't have the emails of the other distros maintainers but the bugs associated with it are:
https://bugzilla.redhat.com/show_bug.cgi?id=1096909
(RHEL7)
https://bugs.launchpad.net/ubuntu/+bug/1319003
(Ubuntu 13.10)
The following distros are affected:
(x) Ubuntu 13.04 and derivatives (3.8)
(v) Ubuntu 13.10 and derivatives (3.11), supported until 2014-07
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) RHEL 7 and derivatives (3.10), expected to be supported until about 2025
(v) openSUSE 13.1 (3.11), expected to be supported until at least 2016-08
(v) SLES 12 (3.12), expected to be supported until about 2024
(v) Mageia 3 (3.8), supported until 2014-11-19
(v) Mageia 4 (3.12), supported until 2015-08-01
(v) Oracle Enterprise Linux with Unbreakable Enterprise Kernel Release 3 (3.8), supported until TBA

Quoted 7 lines

Here is the analysis of the problem and what was put in the RHEL7 bug. The Oracle bug does not exist (as I just backport them in the kernel and send a GIT PULL to Jerry) - but if you would like I can certainly furnish you with one (it would be identical to what is mentioned below).
If you are OK with the backport, I am volunteering Roger and Felipe to assist in jamming^H^H^H^Hbackporting the patches into earlier kernels.

Sure, can you provide backported patches? As-is, they don't apply to the 3.10-stable kernel.

thanks,

greg k-h

Vitaly Kuznetsov

2014-05-20 09:32 UTC #7 lore raw

Quoted 5 lines · Konrad Rzeszutek Wilk

Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:

Quoted 8 lines

bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
I've copied Jerry - the maintainer of the Oracle's kernel. I don't have the emails of the other distros maintainers but the bugs associated with it are:
https://bugzilla.redhat.com/show_bug.cgi?id=1096909
(RHEL7)

I was doing tests with RHEL7 kernel and these patches and unfortunately I see huge performance degradation in some workloads.

I'm in the middle of my testing now but here are some intermediate results.
Test environment:
Fedora-20, xen-4.3.2-2.fc20.x86_64, 3.11.10-301.fc20.x86_64 I do testing with 1-9 RHEL7 PVHVM guests with:
1) Unmodified RHEL7 kernel
2) Only fbe363c476afe8ec992d3baf682670a4bd1b6ce6 applied (revoke foreign access)
3) Both fbe363c476afe8ec992d3baf682670a4bd1b6ce6 and bfe11d6de1c416cea4f3f0f35f864162063ce3fa
(actually 427bfe07e6744c058ce6fc4aa187cda96b635539 is required as well to make build happy, I suggest we backport that to stable as well)

Storage devices are:
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)
2) /tmp/img*.img on tmpfs (persistent grants and indirect descriptors disabled)

Test itself: direct random read with bs=2048k (using fio). (Actually 'dd', 'read/write access', ... show same results)

fio test file:
[fio_read]
ioengine=libaio
blocksize=2048k
rw=randread
filename=/dev/xvdc
randrepeat=1
fallocate=none
direct=1
invalidate=0
runtime=20
time_based

I run fio simultaneously and sum up the result. So, results are:
1) ramdisks: http://hadoop.ru/pubfiles/b1096909_3.11.10_ramdisk.png
2) tmpfiles: http://hadoop.ru/pubfiles/b1096909_3.11.10_tmpfile.png

In few words: patch series has (almost) no effect when persistent grants are enabled (that was expected) and gives me performance regression when persistent grants are disabled (that wasn't expected).

My thoughts are: it seems fbe363c476afe8ec992d3baf682670a4bd1b6ce6 brings performance regression in some cases (at least when persistent grants are disabled). My guess atm is that gnttab_end_foreign_access() (gnttab_end_foreign_access_ref_v1() is being used here) is guilty, for some reason it is looping for some
time. bfe11d6de1c416cea4f3f0f35f864162063ce3fa really brings performance improvement over fbe363c476afe8ec992d3baf682670a4bd1b6ce6 but whole series still brings regression.

I would be glad to hear what could be wrong with my testing in case I'm the only one who sees such behavior. Any other pointers are more than welcome and please feel free to ask for any additional info/testing/whatever from me.

Quoted 72 lines

https://bugs.launchpad.net/ubuntu/+bug/1319003
(Ubuntu 13.10)
The following distros are affected:
(x) Ubuntu 13.04 and derivatives (3.8)
(v) Ubuntu 13.10 and derivatives (3.11), supported until 2014-07
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) RHEL 7 and derivatives (3.10), expected to be supported until about 2025
(v) openSUSE 13.1 (3.11), expected to be supported until at least 2016-08
(v) SLES 12 (3.12), expected to be supported until about 2024
(v) Mageia 3 (3.8), supported until 2014-11-19
(v) Mageia 4 (3.12), supported until 2015-08-01
(v) Oracle Enterprise Linux with Unbreakable Enterprise Kernel Release 3 (3.8), supported until TBA
Here is the analysis of the problem and what was put in the RHEL7 bug. The Oracle bug does not exist (as I just backport them in the kernel and send a GIT PULL to Jerry) - but if you would like I can certainly furnish you with one (it would be identical to what is mentioned below).
If you are OK with the backport, I am volunteering Roger and Felipe to assist in jamming^H^H^H^Hbackporting the patches into earlier kernels.
Summary:
Storage performance regression when Xen backend lacks persistent-grants support
Description of problem:
When used as a Xen guest, RHEL 7 will be slower than older releases in terms s of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). This has been identified and fixed in the 3.13 kernel series, but was not backported to previous LTS kernels due to the nature of the bug (performance only).
While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy operations), adding the copy overhead when the feature is unsupported on
the backend combines the worst of both worlds. This is particularly noticeable
when intensive storage workloads are active from many guests.
How reproducible:
This is always reproducible when a RHEL 7 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants.
Steps to Reproduce:
1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support) - or run it under Amazon EC2
2. Install a set of RHEL 7 guests (which uses kernel 3.10).
3. Measure aggregate storage throughput from all guests.
NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) cannot be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput.
Actual results:
Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12.
Expected results:
Aggregate storage throughput should be at least as good or better if the backend supports persistent grants.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Signature · 1 line

  Vitaly

Vitaly Kuznetsov

2014-05-20 09:33 UTC #16 lore raw

Quoted 5 lines · Konrad Rzeszutek Wilk

Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:

Quoted 8 lines

bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
I've copied Jerry - the maintainer of the Oracle's kernel. I don't have the emails of the other distros maintainers but the bugs associated with it are:
https://bugzilla.redhat.com/show_bug.cgi?id=1096909
(RHEL7)

I was doing tests with RHEL7 kernel and these patches and unfortunately I see huge performance degradation in some workloads.

Storage devices are:
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)
2) /tmp/img*.img on tmpfs (persistent grants and indirect descriptors disabled)

Test itself: direct random read with bs=2048k (using fio). (Actually 'dd', 'read/write access', ... show same results)

fio test file:
[fio_read]
ioengine=libaio
blocksize=2048k
rw=randread
filename=/dev/xvdc
randrepeat=1
fallocate=none
direct=1
invalidate=0
runtime=20
time_based

Quoted 72 lines

https://bugs.launchpad.net/ubuntu/+bug/1319003
(Ubuntu 13.10)
The following distros are affected:
(x) Ubuntu 13.04 and derivatives (3.8)
(v) Ubuntu 13.10 and derivatives (3.11), supported until 2014-07
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) RHEL 7 and derivatives (3.10), expected to be supported until about 2025
(v) openSUSE 13.1 (3.11), expected to be supported until at least 2016-08
(v) SLES 12 (3.12), expected to be supported until about 2024
(v) Mageia 3 (3.8), supported until 2014-11-19
(v) Mageia 4 (3.12), supported until 2015-08-01
(v) Oracle Enterprise Linux with Unbreakable Enterprise Kernel Release 3 (3.8), supported until TBA
Here is the analysis of the problem and what was put in the RHEL7 bug. The Oracle bug does not exist (as I just backport them in the kernel and send a GIT PULL to Jerry) - but if you would like I can certainly furnish you with one (it would be identical to what is mentioned below).
If you are OK with the backport, I am volunteering Roger and Felipe to assist in jamming^H^H^H^Hbackporting the patches into earlier kernels.
Summary:
Storage performance regression when Xen backend lacks persistent-grants support
Description of problem:
When used as a Xen guest, RHEL 7 will be slower than older releases in terms s of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). This has been identified and fixed in the 3.13 kernel series, but was not backported to previous LTS kernels due to the nature of the bug (performance only).
While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy operations), adding the copy overhead when the feature is unsupported on
the backend combines the worst of both worlds. This is particularly noticeable
when intensive storage workloads are active from many guests.
How reproducible:
This is always reproducible when a RHEL 7 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants.
Steps to Reproduce:
1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support) - or run it under Amazon EC2
2. Install a set of RHEL 7 guests (which uses kernel 3.10).
3. Measure aggregate storage throughput from all guests.
NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) cannot be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput.
Actual results:
Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12.
Expected results:
Aggregate storage throughput should be at least as good or better if the backend supports persistent grants.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Signature · 1 line

  Vitaly

Vitaly Kuznetsov

2014-05-20 09:54 UTC #8 lore raw

Quoted 2 lines · Vitaly Kuznetsov

1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)

sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.

Quoted 2 lines

2) /tmp/img*.img on tmpfs (persistent grants and indirect descriptors disabled)

Signature · 1 line

  Vitaly Kuznetsov

Vitaly Kuznetsov

2014-05-20 09:54 UTC #15 lore raw

Quoted 2 lines · Vitaly Kuznetsov

1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)

sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.

Quoted 2 lines

2) /tmp/img*.img on tmpfs (persistent grants and indirect descriptors disabled)

Signature · 1 line

  Vitaly Kuznetsov

Roger Pau Monné

2014-05-20 10:32 UTC #9 lore raw

Quoted 7 lines · Vitaly Kuznetsov

Quoted 2 lines · Vitaly Kuznetsov
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)
sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.

I'm not sure how is that possible, from your description I get that you are using 3.11 on the Dom0, which means blkback has support for persistent grants and indirect descriptors, but the guest is RHEL7, that's using the 3.10 kernel AFAICT, and this kernel only has persistent grants implemented.

Roger.

Roger Pau Monné

2014-05-20 10:32 UTC #14 lore raw

Quoted 7 lines · Vitaly Kuznetsov

Quoted 2 lines · Vitaly Kuznetsov
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)
sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.

Roger.

Vitaly Kuznetsov

2014-05-20 11:41 UTC #10 lore raw

Quoted 6 lines · Roger Pau Monné

Quoted 5 lines · Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)

Quoted 8 lines

Quoted 2 lines
sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.
I'm not sure how is that possible, from your description I get that you are using 3.11 on the Dom0, which means blkback has support for persistent grants and indirect descriptors, but the guest is RHEL7, that's using the 3.10 kernel AFAICT, and this kernel only has persistent grants implemented.

RHEL7 kernel is mostly merged with 3.11 in its Xen part, we have indirect descriptors backported.

Actually I tried my tests with upstream (Fedora) kernel and results were similar. I can try comparing e.g. 3.11.10 with 3.12.0 and provide exact measurements.

Signature · 1 line

  Vitaly

Vitaly Kuznetsov

2014-05-20 11:41 UTC #13 lore raw

Quoted 6 lines · Roger Pau Monné

Quoted 5 lines · Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)

Quoted 8 lines

Quoted 2 lines
sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.
I'm not sure how is that possible, from your description I get that you are using 3.11 on the Dom0, which means blkback has support for persistent grants and indirect descriptors, but the guest is RHEL7, that's using the 3.10 kernel AFAICT, and this kernel only has persistent grants implemented.

RHEL7 kernel is mostly merged with 3.11 in its Xen part, we have indirect descriptors backported.

Actually I tried my tests with upstream (Fedora) kernel and results were similar. I can try comparing e.g. 3.11.10 with 3.12.0 and provide exact measurements.

Signature · 6 lines

  Vitaly

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Felipe Franciosi

2014-05-20 13:59 UTC #11 lore raw

I had a small side-bar thread with Vitaly discussing the comprehensiveness of his measurements and how his tests are being conducted. He will report new results as they become available.

In the meantime, I stand behind that the patches need to be backported and there is a regression if we don't do that.
Ubuntu has already provided a test kernel with the patches pulled in. I will test those as soon as I get the chance (hopefully by the end of the week).
See: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003

Felipe

Quoted 37 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: 20 May 2014 12:41
To: Roger Pau Monne
Cc: Konrad Rzeszutek Wilk; [email protected]; Felipe Franciosi; [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]
Subject: Re: [Xen-devel] Backport request to stable of two performance related fixes for xen-blkfront (3.13 fixes to earlier trees)
Quoted 6 lines · Roger Pau Monné
Quoted 5 lines · Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)
Quoted 8 lines
Quoted 2 lines
sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.
I'm not sure how is that possible, from your description I get that you are using 3.11 on the Dom0, which means blkback has support for persistent grants and indirect descriptors, but the guest is RHEL7, that's using the 3.10 kernel AFAICT, and this kernel only has persistent grants implemented.
RHEL7 kernel is mostly merged with 3.11 in its Xen part, we have indirect descriptors backported.
Actually I tried my tests with upstream (Fedora) kernel and results were similar. I can try comparing e.g. 3.11.10 with 3.12.0 and provide exact measurements.
--
Vitaly

Vitaly Kuznetsov

2014-05-22 08:52 UTC #12 lore raw

Quoted 3 lines · Felipe Franciosi

I had a small side-bar thread with Vitaly discussing the comprehensiveness of his measurements and how his tests are being conducted. He will report new results as they become available.

I'm back ;-)

In short: I think I was able to find a very 'special' case when this patch series leads to io performance regression. It is clearly visible with upstream kernel, no RHEL specifics involved. Now the details.

I compare IO performance with 3 kernel
1) "unpatched_upstream" means unmodified Linus's
60b5f90d0fac7585f1a43ccdad06787b97eda0ab build

2) "revertall_upstream" means upstream with all patches reverted (60b5f90d0fac7585f1a43ccdad06787b97eda0ab + 3 commits reverted:
427bfe07e6744c058ce6fc4aa187cda96b635539, bfe11d6de1c416cea4f3f0f35f864162063ce3fa, fbe363c476afe8ec992d3baf682670a4bd1b6ce6)

3) "revokefaonly_upstream" means upstream with "revoke foreign access" patch only (60b5f90d0fac7585f1a43ccdad06787b97eda0ab + 2 commits revered:
427bfe07e6744c058ce6fc4aa187cda96b635539, bfe11d6de1c416cea4f3f0f35f864162063ce3fa)

I have the following setup:
1) Single-cpu "Intel(R) Xeon(R)CPU W3550 @ 3.07GHz" system, 4 cores
2) No hyper threading, turbo boost, ..
3) Dom0 is running 3.11.10-301.fc20.x86_64 kernel, xen-4.3.2-3.fc20.x86_64
4) Dom0 is pinned to Core0
5) I create 9 clients pinned to (Core1, Core2, Core3, Core1, Core2, Core3, Core1, Core2, Core3)
6) Clients are identical, the only thing which differs is kernel.

Now the most important part.
1) For each client I create 1G file on tmpfs in dom0
2) I attach these files to clients as e.g. "file:/tmp/img10.img,xvdc,rw" (no blkback)
3) In clients I see these devices as "blkfront: xvdc: barrier: enabled; persistent grants: disabled; indirect descriptors: disabled;"

Tests:
I run fio simultaneously on 1 - 9 clients and measure aggregate throughput. Each test is being run 3 times and average is being taken. I run tests with different BS (4k, 64k, 512k, 2048k) and different RW (randread, randrw)
Fio job:
[fio_jobname]
ioengine=libaio
blocksize=<BS>
filename=/dev/xvdc
randrepeat=1
fallocate=none
direct=1
invalidate=0
runtime=20
time_based
rw=<randread|randrw>

Now the results:

rw=randread:
1) 4k (strange): http://hadoop.ru/pubfiles/bug1096909/4k_r.png
2) 64k: http://hadoop.ru/pubfiles/bug1096909/64k_r.png
3) 512k: http://hadoop.ru/pubfiles/bug1096909/512k_r.png
4) 2048k: http://hadoop.ru/pubfiles/bug1096909/2048k_r.png

rw=randrw:
1) 4k (strange): http://hadoop.ru/pubfiles/bug1096909/4k_rw.png
2) 64k: http://hadoop.ru/pubfiles/bug1096909/64k_rw.png
3) 512k: http://hadoop.ru/pubfiles/bug1096909/512k_rw.png
4) 2048k: http://hadoop.ru/pubfiles/bug1096909/2048k_rw.png

In short, 'revertall_upstream' wins everywhere with significant-enough difference.

P.S. To see the regression such complicated setup is not required, it is clearly visible even with 1 client. E.g.:

# for q in `seq 1 10`; do fio --minimal --client <client_with_"revertall_upstream"_kernel> test_r.fio | grep READ; done
   READ: io=8674.0MB, aggrb=444086KB/s, minb=444086KB/s, maxb=444086KB/s, mint=20001msec, maxt=20001msec
   READ: io=8626.0MB, aggrb=441607KB/s, minb=441607KB/s, maxb=441607KB/s, mint=20002msec, maxt=20002msec
   READ: io=8620.0MB, aggrb=441277KB/s, minb=441277KB/s, maxb=441277KB/s, mint=20003msec, maxt=20003msec
   READ: io=8522.0MB, aggrb=436304KB/s, minb=436304KB/s, maxb=436304KB/s, mint=20001msec, maxt=20001msec
   READ: io=8218.0MB, aggrb=420698KB/s, minb=420698KB/s, maxb=420698KB/s, mint=20003msec, maxt=20003msec
   READ: io=8374.0MB, aggrb=428705KB/s, minb=428705KB/s, maxb=428705KB/s, mint=20002msec, maxt=20002msec
   READ: io=8198.0MB, aggrb=419653KB/s, minb=419653KB/s, maxb=419653KB/s, mint=20004msec, maxt=20004msec
   READ: io=7586.0MB, aggrb=388306KB/s, minb=388306KB/s, maxb=388306KB/s, mint=20005msec, maxt=20005msec
   READ: io=8512.0MB, aggrb=435749KB/s, minb=435749KB/s, maxb=435749KB/s, mint=20003msec, maxt=20003msec
   READ: io=8524.0MB, aggrb=436319KB/s, minb=436319KB/s, maxb=436319KB/s, mint=20005msec, maxt=20005msec

# for q in `seq 1 10`; do fio --minimal --client <client_with_"unpatched_upstream"_kernel> test_r.fio | grep READ; done
   READ: io=7236.0MB, aggrb=370464KB/s, minb=370464KB/s, maxb=370464KB/s, mint=20001msec, maxt=20001msec
   READ: io=6506.0MB, aggrb=333090KB/s, minb=333090KB/s, maxb=333090KB/s, mint=20001msec, maxt=20001msec
   READ: io=6584.0MB, aggrb=337050KB/s, minb=337050KB/s, maxb=337050KB/s, mint=20003msec, maxt=20003msec
   READ: io=7120.0MB, aggrb=364489KB/s, minb=364489KB/s, maxb=364489KB/s, mint=20003msec, maxt=20003msec
   READ: io=6610.0MB, aggrb=338347KB/s, minb=338347KB/s, maxb=338347KB/s, mint=20005msec, maxt=20005msec
   READ: io=7024.0MB, aggrb=359556KB/s, minb=359556KB/s, maxb=359556KB/s, mint=20004msec, maxt=20004msec
   READ: io=7320.0MB, aggrb=374765KB/s, minb=374765KB/s, maxb=374765KB/s, mint=20001msec, maxt=20001msec
   READ: io=6540.0MB, aggrb=334814KB/s, minb=334814KB/s, maxb=334814KB/s, mint=20002msec, maxt=20002msec
   READ: io=6636.0MB, aggrb=339661KB/s, minb=339661KB/s, maxb=339661KB/s, mint=20006msec, maxt=20006msec
   READ: io=6594.0MB, aggrb=337595KB/s, minb=337595KB/s, maxb=337595KB/s, mint=20001msec, maxt=20001msec

Dumb 'dd' test shows the same:
"revertall_upstream" client:

# time for ntry in `seq 1 100`; do dd if=/dev/xvdc of=/dev/null bs=2048k 2> /dev/null; done

real0m16.262s
user0m0.189s
sys0m7.021s

"unpatched_upstream"

# time for ntry in `seq 1 100`; do dd if=/dev/xvdc of=/dev/null bs=2048k 2> /dev/null; done

real0m19.938s
user0m0.174s
sys0m9.489s

I tried running newer Dom0 (3.14.4-200.fc20.x86_64) but that makes no difference.

P.P.S. I understand this test differs a lot from what these patches were supposed to fix and I'm not trying to say 'no' for stable backport, but I also thinks this test data can be interesting as well.

And thanks, Felipe, for all your hardware hints!

Quoted 48 lines

In the meantime, I stand behind that the patches need to be backported and there is a regression if we don't do that.
Ubuntu has already provided a test kernel with the patches pulled in. I will test those as soon as I get the chance (hopefully by the end of the week).
See: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003
Felipe
Quoted 31 lines
-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: 20 May 2014 12:41
To: Roger Pau Monne
Cc: Konrad Rzeszutek Wilk; [email protected]; Felipe Franciosi; [email protected]; [email protected]; [email protected]; [email protected]; xen- [email protected]
Subject: Re: [Xen-devel] Backport request to stable of two performance related fixes for xen-blkfront (3.13 fixes to earlier trees)
Quoted 6 lines · Roger Pau Monné
Quoted 5 lines · Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov
1) ramdisks (/dev/ram*) (persistent grants and indirect descriptors disabled)
Quoted 8 lines
Quoted 2 lines
sorry, there was a typo. persistent grants and indirect descriptors are enabled with ramdisks, otherwise such testing won't make any sense.
I'm not sure how is that possible, from your description I get that you are using 3.11 on the Dom0, which means blkback has support for persistent grants and indirect descriptors, but the guest is RHEL7, that's using the 3.10 kernel AFAICT, and this kernel only has persistent grants implemented.
RHEL7 kernel is mostly merged with 3.11 in its Xen part, we have indirect descriptors backported.
Quoted 6 lines
Actually I tried my tests with upstream (Fedora) kernel and results were similar. I can try comparing e.g. 3.11.10 with 3.12.0 and provide exact measurements.
--
Vitaly
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Signature · 1 line

  Vitaly

Roger Pau Monné

Reviewed-by David Vrabel Acked-by Matt Wilson

2014-05-22 12:54 UTC #4 lore raw

Quoted 37 lines · Greg KH

Quoted 36 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
I've copied Jerry - the maintainer of the Oracle's kernel. I don't have the emails of the other distros maintainers but the bugs associated with it are:
https://bugzilla.redhat.com/show_bug.cgi?id=1096909
(RHEL7)
https://bugs.launchpad.net/ubuntu/+bug/1319003
(Ubuntu 13.10)
The following distros are affected:
(x) Ubuntu 13.04 and derivatives (3.8)
(v) Ubuntu 13.10 and derivatives (3.11), supported until 2014-07
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) RHEL 7 and derivatives (3.10), expected to be supported until about 2025
(v) openSUSE 13.1 (3.11), expected to be supported until at least 2016-08
(v) SLES 12 (3.12), expected to be supported until about 2024
(v) Mageia 3 (3.8), supported until 2014-11-19
(v) Mageia 4 (3.12), supported until 2015-08-01
(v) Oracle Enterprise Linux with Unbreakable Enterprise Kernel Release 3 (3.8), supported until TBA
Here is the analysis of the problem and what was put in the RHEL7 bug. The Oracle bug does not exist (as I just backport them in the kernel and send a GIT PULL to Jerry) - but if you would like I can certainly furnish you with one (it would be identical to what is mentioned below).

Quoted 5 lines

Quoted 2 lines
If you are OK with the backport, I am volunteering Roger and Felipe to assist in jamming^H^H^H^Hbackporting the patches into earlier kernels.
Sure, can you provide backported patches? As-is, they don't apply to the 3.10-stable kernel.

Here are the backported patches to 3.10 stable, I would like however to get some testing/benchmarking on them before applying, since it's not a trivial backport. Could you please give them a spin Felipe?

Roger.

Quoted 1 line

From 8db2b1bff4b54dc01592f9e14b9dae32c341f435 Mon Sep 17 00:00:00 2001

From: Roger Pau Monne <[email protected]>
Date: Wed, 21 May 2014 16:51:47 +0200
Subject: [PATCH 1/2] backport of fbe363 to stable 3.10.y branch
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

xen-blkfront: revoke foreign access for grants not mapped by the backend

There's no need to keep the foreign access in a grant if it is not persistently mapped by the backend. This allows us to free grants that are not mapped by the backend, thus preventing blkfront from hoarding all grants.

The main effect of this is that blkfront will only persistently map the same grants as the backend, and it will always try to use grants that are already mapped by the backend. Also the number of persistent grants in blkfront is the same as in blkback (and is controlled by the value in blkback).

Reviewed-by David Vrabel <[email protected]>

Acked-by Matt Wilson <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc David Vrabel <[email protected]>

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

Signed-off-by Jens Axboe <[email protected]>

---
 drivers/block/xen-blkfront.c |   24 +++++++++++++++++++++---
 1 files changed, 21 insertions(+), 3 deletions(-)

drivers/block/xen-blkfront.c +21 -3

index 1735b0d..cbd9f6b 100644
@@ -894,9 +894,27 @@ static void blkif_completion(struct blk_shadow *s, struct blkfront_info *info,
 		}
 	}
 	/* Add the persistent grant into the list of free grants */
-	for (i = 0; i < s->req.u.rw.nr_segments; i++) {
-		list_add(&s->grants_used[i]->node, &info->persistent_gnts);
-		info->persistent_gnts_c++;
+	for (i = 0; i < nseg; i++) {
+		if (gnttab_query_foreign_access(s->grants_used[i]->gref)) {
+			/*
+			 * If the grant is still mapped by the backend (the
+			 * backend has chosen to make this grant persistent)
+			 * we add it at the head of the list, so it will be
+			 * reused first.
+			 */
+			list_add(&s->grants_used[i]->node, &info->persistent_gnts);
+			info->persistent_gnts_c++;
+		} else {
+			/*
+			 * If the grant is not mapped by the backend we end the
+			 * foreign access and add it to the tail of the list,
+			 * so it will not be picked again unless we run out of
+			 * persistent grants.
+			 */
+			gnttab_end_foreign_access(s->grants_used[i]->gref, 0, 0UL);
+			s->grants_used[i]->gref = GRANT_INVALID_REF;
+			list_add_tail(&s->grants_used[i]->node, &info->persistent_gnts);
+		}
 	}
 }

Signature · 1 line

1.7.7.5 (Apple Git-26)

Quoted 1 line

From 0cb3ce2b6838aa9a28b81dacf497a4ae11610826 Mon Sep 17 00:00:00 2001

From: Roger Pau Monne <[email protected]>
Date: Wed, 21 May 2014 16:57:58 +0200
Subject: [PATCH 2/2] backport of bfe11d to stable 3.10.y branch
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

xen-blkfront: restore the non-persistent data path

When persistent grants were added they were always used, even if the backend doesn't have this feature (there's no harm in always using the same set of pages). This restores the old data path when the backend doesn't have persistent grants, removing the burden of doing a memcpy when it is not actually needed.

Reported-by Felipe Franciosi <[email protected]>

Cc Felipe Franciosi <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc David Vrabel <[email protected]>

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

[v2: Fix up whitespace issues]

---
 drivers/block/xen-blkfront.c |  105 +++++++++++++++++++++++++++++------------
 1 files changed, 74 insertions(+), 31 deletions(-)

drivers/block/xen-blkfront.c +74 ~~-31~~

229 lines not on this page · raw message

Signature · 1 line

1.7.7.5 (Apple Git-26)

Roger Pau Monné

Reviewed-by David Vrabel Acked-by Matt Wilson

2014-05-22 12:54 UTC #6 lore raw

Quoted 37 lines · Greg KH

Quoted 36 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
I've copied Jerry - the maintainer of the Oracle's kernel. I don't have the emails of the other distros maintainers but the bugs associated with it are:
https://bugzilla.redhat.com/show_bug.cgi?id=1096909
(RHEL7)
https://bugs.launchpad.net/ubuntu/+bug/1319003
(Ubuntu 13.10)
The following distros are affected:
(x) Ubuntu 13.04 and derivatives (3.8)
(v) Ubuntu 13.10 and derivatives (3.11), supported until 2014-07
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) RHEL 7 and derivatives (3.10), expected to be supported until about 2025
(v) openSUSE 13.1 (3.11), expected to be supported until at least 2016-08
(v) SLES 12 (3.12), expected to be supported until about 2024
(v) Mageia 3 (3.8), supported until 2014-11-19
(v) Mageia 4 (3.12), supported until 2015-08-01
(v) Oracle Enterprise Linux with Unbreakable Enterprise Kernel Release 3 (3.8), supported until TBA
Here is the analysis of the problem and what was put in the RHEL7 bug. The Oracle bug does not exist (as I just backport them in the kernel and send a GIT PULL to Jerry) - but if you would like I can certainly furnish you with one (it would be identical to what is mentioned below).

Quoted 5 lines

Quoted 2 lines
If you are OK with the backport, I am volunteering Roger and Felipe to assist in jamming^H^H^H^Hbackporting the patches into earlier kernels.
Sure, can you provide backported patches? As-is, they don't apply to the 3.10-stable kernel.

Roger.

Quoted 1 line

From 8db2b1bff4b54dc01592f9e14b9dae32c341f435 Mon Sep 17 00:00:00 2001

xen-blkfront: revoke foreign access for grants not mapped by the backend

Reviewed-by David Vrabel <[email protected]>

Acked-by Matt Wilson <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc David Vrabel <[email protected]>

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

Signed-off-by Jens Axboe <[email protected]>

---
 drivers/block/xen-blkfront.c |   24 +++++++++++++++++++++---
 1 files changed, 21 insertions(+), 3 deletions(-)

drivers/block/xen-blkfront.c +21 -3

index 1735b0d..cbd9f6b 100644
@@ -894,9 +894,27 @@ static void blkif_completion(struct blk_shadow *s, struct blkfront_info *info,
 		}
 	}
 	/* Add the persistent grant into the list of free grants */
-	for (i = 0; i < s->req.u.rw.nr_segments; i++) {
-		list_add(&s->grants_used[i]->node, &info->persistent_gnts);
-		info->persistent_gnts_c++;
+	for (i = 0; i < nseg; i++) {
+		if (gnttab_query_foreign_access(s->grants_used[i]->gref)) {
+			/*
+			 * If the grant is still mapped by the backend (the
+			 * backend has chosen to make this grant persistent)
+			 * we add it at the head of the list, so it will be
+			 * reused first.
+			 */
+			list_add(&s->grants_used[i]->node, &info->persistent_gnts);
+			info->persistent_gnts_c++;
+		} else {
+			/*
+			 * If the grant is not mapped by the backend we end the
+			 * foreign access and add it to the tail of the list,
+			 * so it will not be picked again unless we run out of
+			 * persistent grants.
+			 */
+			gnttab_end_foreign_access(s->grants_used[i]->gref, 0, 0UL);
+			s->grants_used[i]->gref = GRANT_INVALID_REF;
+			list_add_tail(&s->grants_used[i]->node, &info->persistent_gnts);
+		}
 	}
 }

Signature · 1 line

1.7.7.5 (Apple Git-26)

Quoted 1 line

From 0cb3ce2b6838aa9a28b81dacf497a4ae11610826 Mon Sep 17 00:00:00 2001

xen-blkfront: restore the non-persistent data path

Reported-by Felipe Franciosi <[email protected]>

Cc Felipe Franciosi <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc David Vrabel <[email protected]>

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

[v2: Fix up whitespace issues]

---
 drivers/block/xen-blkfront.c |  105 +++++++++++++++++++++++++++++------------
 1 files changed, 74 insertions(+), 31 deletions(-)

drivers/block/xen-blkfront.c +74 ~~-31~~

229 lines not on this page · raw message

Signature · 1 line

1.7.7.5 (Apple Git-26)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Felipe Franciosi

2014-06-02 07:14 UTC #5 lore raw

Quoted 57 lines

-----Original Message-----
From: Roger Pau Monné [mailto:[email protected]]
Sent: 22 May 2014 13:54
To: Greg KH; Konrad Rzeszutek Wilk
Cc: [email protected]; [email protected]; xen- [email protected]; Felipe Franciosi; [email protected]; [email protected]
Subject: Re: Backport request to stable of two performance related fixes for xen-blkfront (3.13 fixes to earlier trees)
Quoted 11 lines · Greg KH
Quoted 10 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
I've copied Jerry - the maintainer of the Oracle's kernel. I don't have the emails of the other distros maintainers but the bugs associated
with it are:
Quoted 34 lines
Quoted 33 lines
https://bugzilla.redhat.com/show_bug.cgi?id=1096909
(RHEL7)
https://bugs.launchpad.net/ubuntu/+bug/1319003
(Ubuntu 13.10)
The following distros are affected:
(x) Ubuntu 13.04 and derivatives (3.8)
(v) Ubuntu 13.10 and derivatives (3.11), supported until 2014-07
(x) Fedora 17 (3.8 and 3.9 in updates)
(x) Fedora 18 (3.8, 3.9, 3.10, 3.11 in updates)
(v) Fedora 19 (3.9; 3.10, 3.11, 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) Fedora 20 (3.11; 3.12 in updates; fixed with latest update to 3.13), supported until TBA
(v) RHEL 7 and derivatives (3.10), expected to be supported until about 2025
(v) openSUSE 13.1 (3.11), expected to be supported until at least 2016-08
(v) SLES 12 (3.12), expected to be supported until about 2024
(v) Mageia 3 (3.8), supported until 2014-11-19
(v) Mageia 4 (3.12), supported until 2015-08-01
(v) Oracle Enterprise Linux with Unbreakable Enterprise Kernel Release 3 (3.8), supported until TBA
Here is the analysis of the problem and what was put in the RHEL7 bug. The Oracle bug does not exist (as I just backport them in the kernel and send a GIT PULL to Jerry) - but if you would like I can certainly furnish you with one (it would be identical to what is mentioned below).
If you are OK with the backport, I am volunteering Roger and Felipe to assist in jamming^H^H^H^Hbackporting the patches into earlier kernels.

Quoted 6 lines

Quoted 2 lines
Sure, can you provide backported patches? As-is, they don't apply to the 3.10-stable kernel.
Here are the backported patches to 3.10 stable, I would like however to get some testing/benchmarking on them before applying, since it's not a trivial backport. Could you please give them a spin Felipe?

Apologies for the delay in remeasuring this.

I can confirm the backport drastically improves aggregate throughput (at least) when the backend does not support persistent grants. This is very visible with moderate-sized requests and more than one VM (on my graphs, above four guests). Naturally, the magnitude of the regression will inevitably vary depending on the host's characteristics.

The attached graphs were measured on XenServer Creedence #85466 which uses tapdisk3 (grant copy). They are the result of a single run, so some variation is expected.

I used Ubuntu 13.10 64 bit guests and installed 3.10.40. I used 5 SSDs (4 Microns and 1 Fusion-io). Each SSD had 10 LVM Logical Volumes, one assigned to each VM (therefore each VM had 5 extra virtual disks). The benchmark was sequential reads using the specified block size directly on the virtual block device with O_DIRECT for a duration of 10s each, reporting the total throughput. It was synchronised so that the specified number of guests executed the IO operations at the same time.

The only difference between the two tests are Roger's patches. Please see attached.

Quoted 2 lines

Roger.

Cheers,
Felipe

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Greg KH

maintainer

2014-06-04 05:48 UTC #17 lore raw

Quoted 7 lines · Konrad Rzeszutek Wilk

Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6

Now queued up, thanks.

greg k-h

Greg KH

maintainer

2014-06-04 05:48 UTC #35 lore raw

Quoted 7 lines · Konrad Rzeszutek Wilk

Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6

Now queued up, thanks.

greg k-h

Jiri Slaby

2014-06-06 10:47 UTC #18 lore raw

Quoted 10 lines · Greg KH

Quoted 7 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.

AFAIU, they introduce a performance regression.

Vitaly?

Signature · 2 lines

js
suse labs

Jiri Slaby

2014-06-06 10:47 UTC #34 lore raw

Quoted 10 lines · Greg KH

Quoted 7 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.

AFAIU, they introduce a performance regression.

Vitaly?

Signature · 2 lines

js
suse labs

Vitaly Kuznetsov

2014-06-06 10:58 UTC #19 lore raw

Quoted 7 lines · Jiri Slaby

Quoted 6 lines · Greg KH
Quoted 5 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.
Vitaly?

I'm aware of a performance regression in a 'very special' case when ramdisks or files on tmpfs are being used as storage, I post my results a while ago:
https://lkml.org/lkml/2014/5/22/164
I'm not sure if that 'special' case requires investigation and/or should prevent us from doing stable backport but it would be nice if someone tries to reproduce it at least.

I'm going to make a bunch of tests with FusionIO drives and sequential read to replicate same test Felipe did, I'll report as soon as I have data (beginning of next week hopefuly).

Signature · 1 line

  Vitaly

Vitaly Kuznetsov

2014-06-06 10:58 UTC #27 lore raw

Quoted 7 lines · Jiri Slaby

Quoted 6 lines · Greg KH
Quoted 5 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.
Vitaly?

I'm going to make a bunch of tests with FusionIO drives and sequential read to replicate same test Felipe did, I'll report as soon as I have data (beginning of next week hopefuly).

Signature · 1 line

  Vitaly

Greg KH

maintainer

2014-06-06 13:56 UTC #28 lore raw

Quoted 7 lines · Jiri Slaby

Quoted 6 lines · Greg KH
Quoted 5 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.

That "regression" is also in mainline, right? As Konrad doesn't seem to think it matters, I'm deferring to the maintainer here.

thanks,

greg k-h

Greg KH

maintainer

2014-06-06 13:57 UTC #33 lore raw

Quoted 7 lines · Jiri Slaby

Quoted 6 lines · Greg KH
Quoted 5 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.

That "regression" is also in mainline, right? As Konrad doesn't seem to think it matters, I'm deferring to the maintainer here.

thanks,

greg k-h

Konrad Rzeszutek Wilk

author

2014-06-06 14:02 UTC #29 lore raw

Quoted 11 lines · Greg KH

Quoted 10 lines · Jiri Slaby
Quoted 9 lines · Greg KH
Quoted 7 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
Now queued up, thanks.
AFAIU, they introduce a performance regression.
That "regression" is also in mainline, right? As Konrad doesn't seem to think it matters, I'm deferring to the maintainer here.

Hehe.

Greg is correct - the performance regression with tmpfs/ramfs does exist upstream and once a fix has been established will be dealt with. Right now we are fousing on the 99% usage models which is solid state, rotational, and flash (just got one of those) and the two patches outlined above are needed for the stable trees.

Thank you.

Hopefully I haven't confused the issue here.

Quoted 4 lines

thanks,
greg k-h

Konrad Rzeszutek Wilk

author

2014-06-06 14:02 UTC #32 lore raw

Quoted 11 lines · Greg KH

Quoted 10 lines · Jiri Slaby
Quoted 9 lines · Greg KH
Quoted 7 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
Now queued up, thanks.
AFAIU, they introduce a performance regression.
That "regression" is also in mainline, right? As Konrad doesn't seem to think it matters, I'm deferring to the maintainer here.

Hehe.

Thank you.

Hopefully I haven't confused the issue here.

Quoted 4 lines

thanks,
greg k-h

Jiri Slaby

2014-06-06 14:03 UTC #30 lore raw

Quoted 19 lines · Konrad Rzeszutek Wilk

Quoted 11 lines · Greg KH
Quoted 10 lines · Jiri Slaby
Quoted 9 lines · Greg KH
Quoted 7 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Now queued up, thanks.
AFAIU, they introduce a performance regression.
That "regression" is also in mainline, right? As Konrad doesn't seem to think it matters, I'm deferring to the maintainer here.

Quoted 7 lines

Hehe.
Greg is correct - the performance regression with tmpfs/ramfs does exist upstream and once a fix has been established will be dealt with. Right now we are fousing on the 99% usage models which is solid state, rotational, and flash (just got one of those) and the two patches outlined above are needed for the stable trees.

Ok, I wanted to be sure before I take these to 3.12.

Thanks.

Signature · 2 lines

js
suse labs

Jiri Slaby

2014-06-06 14:04 UTC #31 lore raw

Quoted 19 lines · Konrad Rzeszutek Wilk

Quoted 11 lines · Greg KH
Quoted 10 lines · Jiri Slaby
Quoted 9 lines · Greg KH
Quoted 7 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Now queued up, thanks.
AFAIU, they introduce a performance regression.
That "regression" is also in mainline, right? As Konrad doesn't seem to think it matters, I'm deferring to the maintainer here.

Quoted 7 lines

Hehe.
Greg is correct - the performance regression with tmpfs/ramfs does exist upstream and once a fix has been established will be dealt with. Right now we are fousing on the 99% usage models which is solid state, rotational, and flash (just got one of those) and the two patches outlined above are needed for the stable trees.

Ok, I wanted to be sure before I take these to 3.12.

Thanks.

Signature · 2 lines

js
suse labs

Vitaly Kuznetsov

2014-06-10 13:19 UTC #20 lore raw

Quoted 26 lines · Vitaly Kuznetsov

Quoted 7 lines · Jiri Slaby
Quoted 6 lines · Greg KH
Quoted 5 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.
Vitaly?
I'm aware of a performance regression in a 'very special' case when ramdisks or files on tmpfs are being used as storage, I post my results a while ago:
https://lkml.org/lkml/2014/5/22/164
I'm not sure if that 'special' case requires investigation and/or should prevent us from doing stable backport but it would be nice if someone tries to reproduce it at least.

Quoted 3 lines

I'm going to make a bunch of tests with FusionIO drives and sequential read to replicate same test Felipe did, I'll report as soon as I have data (beginning of next week hopefuly).

Turns out the regression I'm observing with these patches is not restricted to tmpfs/ramdisk usage.

I was doing tests with Fusion-io ioDrive Duo 320GB (Dual Adapter) on HP ProLiant DL380 G6 (2xE5540, 8G RAM). Hyperthreading is disabled, Dom0 is pinned to CPU0 (cores 0,1,2,3) I run up to 8 guests with 1 vCPU each, they are pinned to CPU1 (cores 4,5,6,7,4,5,6,7). I tried differed pinning (Dom0 to 0,1,4,5, DomUs to 2,3,6,7,2,3,6,7 to balance NUMA, that doesn't make any difference to the results). I was testing on top of Xen-4.3.2.

I was testing two storage configurations:
1) Plain 10G partitions from one Fusion drive (/dev/fioa) are attached to guests
2) LVM group is created on top of both drives (/dev/fioa, /dev/fiob), 10G logical volumes are created with striping (lvcreate -i2 ...)

Test is done by simultaneous fio run in guests (rw=read, direct=1) for 10 second. Each test was performed 3 times and the average was taken. Kernels I compare are:
1) v3.15-rc5-157-g60b5f90 unmodified
2) v3.15-rc5-157-g60b5f90 with 427bfe07e6744c058ce6fc4aa187cda96b635539, bfe11d6de1c416cea4f3f0f35f864162063ce3fa, and fbe363c476afe8ec992d3baf682670a4bd1b6ce6 reverted.

First test was done with Dom0 with persistent grant support (Fedora's 3.14.4-200.fc20.x86_64):
1) Partitions:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_pgrants_partitions.png
(same markers mean same bs, we get 860 MB/s here, patches make no difference, result matches expectation)

2) LVM Stripe:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_pgrants_stripe.png
(1715 MB/s, patches make no difference, result matches expectation)

Second test was performed with Dom0 without persistent grants support (Fedora's 3.7.9-205.fc18.x86_64)
1) Partitions:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_nopgrants_partitions.png
(860 MB/sec again, patches worsen a bit overall throughput with 1-3 clients)

2) LVM Stripe:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_nopgrants_stripe.png
(Here we see the same regression I observed with ramdisks and tmpfs files, unmodified kernel: 1550MB/s, with patches reverted: 1715MB/s).

The only major difference with Felipe's test is that he was using blktap3 with XenServer and I'm using standard blktap2.

Signature · 1 line

  Vitaly

Roger Pau Monné

2014-06-10 16:55 UTC #21 lore raw

Quoted 75 lines · Vitaly Kuznetsov

Quoted 26 lines · Vitaly Kuznetsov
Quoted 7 lines · Jiri Slaby
Quoted 6 lines · Greg KH
Quoted 5 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.
Vitaly?
I'm aware of a performance regression in a 'very special' case when ramdisks or files on tmpfs are being used as storage, I post my results a while ago:
https://lkml.org/lkml/2014/5/22/164
I'm not sure if that 'special' case requires investigation and/or should prevent us from doing stable backport but it would be nice if someone tries to reproduce it at least.
Quoted 3 lines
I'm going to make a bunch of tests with FusionIO drives and sequential read to replicate same test Felipe did, I'll report as soon as I have data (beginning of next week hopefuly).
Turns out the regression I'm observing with these patches is not restricted to tmpfs/ramdisk usage.
I was doing tests with Fusion-io ioDrive Duo 320GB (Dual Adapter) on HP ProLiant DL380 G6 (2xE5540, 8G RAM). Hyperthreading is disabled, Dom0 is pinned to CPU0 (cores 0,1,2,3) I run up to 8 guests with 1 vCPU each, they are pinned to CPU1 (cores 4,5,6,7,4,5,6,7). I tried differed pinning (Dom0 to 0,1,4,5, DomUs to 2,3,6,7,2,3,6,7 to balance NUMA, that doesn't make any difference to the results). I was testing on top of Xen-4.3.2.
I was testing two storage configurations:
1) Plain 10G partitions from one Fusion drive (/dev/fioa) are attached to guests
2) LVM group is created on top of both drives (/dev/fioa, /dev/fiob), 10G logical volumes are created with striping (lvcreate -i2 ...)
Test is done by simultaneous fio run in guests (rw=read, direct=1) for 10 second. Each test was performed 3 times and the average was taken. Kernels I compare are:
1) v3.15-rc5-157-g60b5f90 unmodified
2) v3.15-rc5-157-g60b5f90 with 427bfe07e6744c058ce6fc4aa187cda96b635539, bfe11d6de1c416cea4f3f0f35f864162063ce3fa, and fbe363c476afe8ec992d3baf682670a4bd1b6ce6 reverted.
First test was done with Dom0 with persistent grant support (Fedora's 3.14.4-200.fc20.x86_64):
1) Partitions:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_pgrants_partitions.png
(same markers mean same bs, we get 860 MB/s here, patches make no difference, result matches expectation)
2) LVM Stripe:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_pgrants_stripe.png
(1715 MB/s, patches make no difference, result matches expectation)
Second test was performed with Dom0 without persistent grants support (Fedora's 3.7.9-205.fc18.x86_64)
1) Partitions:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_nopgrants_partitions.png
(860 MB/sec again, patches worsen a bit overall throughput with 1-3 clients)

Quoted 7 lines

2) LVM Stripe:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_nopgrants_stripe.png
(Here we see the same regression I observed with ramdisks and tmpfs files, unmodified kernel: 1550MB/s, with patches reverted: 1715MB/s).
The only major difference with Felipe's test is that he was using blktap3 with XenServer and I'm using standard blktap2.

Hello,

I don't think you are using blktap2, I guess you are using blkback. Also, running the test only for 10s and 3 repetitions seems too low, I would probably try to run the tests for a longer time and do more repetitions, and include the standard deviation also.

Could you try to revert the patches independently to see if it's a specific commit that introduces the regression?

Thanks, Roger.

Felipe Franciosi

2014-06-10 17:26 UTC #26 lore raw

Quoted 79 lines · Vitaly Kuznetsov

Quoted 26 lines · Vitaly Kuznetsov
Quoted 7 lines · Jiri Slaby
Quoted 7 lines · Greg KH
Quoted 6 lines · Konrad Rzeszutek Wilk
Hey Greg
This email is in regards to backporting two patches to stable that fall under the 'performance' rule:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
bfe11d6de1c416cea4f3f0f35f864162063ce3fa
fbe363c476afe8ec992d3baf682670a4bd1b6ce6
Now queued up, thanks.
AFAIU, they introduce a performance regression.
Vitaly?
I'm aware of a performance regression in a 'very special' case when ramdisks or files on tmpfs are being used as storage, I post my results a while ago:
https://lkml.org/lkml/2014/5/22/164
I'm not sure if that 'special' case requires investigation and/or should prevent us from doing stable backport but it would be nice if someone tries to reproduce it at least.
Quoted 3 lines
I'm going to make a bunch of tests with FusionIO drives and sequential read to replicate same test Felipe did, I'll report as soon as I have data (beginning of next week hopefuly).
Turns out the regression I'm observing with these patches is not restricted to tmpfs/ramdisk usage.
I was doing tests with Fusion-io ioDrive Duo 320GB (Dual Adapter) on HP ProLiant DL380 G6 (2xE5540, 8G RAM). Hyperthreading is disabled, Dom0 is pinned to CPU0 (cores 0,1,2,3) I run up to 8 guests with 1 vCPU each, they are pinned to CPU1 (cores 4,5,6,7,4,5,6,7). I tried differed pinning (Dom0 to 0,1,4,5, DomUs to 2,3,6,7,2,3,6,7 to balance NUMA, that doesn't make any difference to the results). I was testing on top of Xen-4.3.2.
I was testing two storage configurations:
1) Plain 10G partitions from one Fusion drive (/dev/fioa) are attached to guests
2) LVM group is created on top of both drives (/dev/fioa, /dev/fiob), 10G logical volumes are created with striping (lvcreate -i2 ...)
Test is done by simultaneous fio run in guests (rw=read, direct=1) for 10 second. Each test was performed 3 times and the average was taken. Kernels I compare are:
1) v3.15-rc5-157-g60b5f90 unmodified
2) v3.15-rc5-157-g60b5f90 with 427bfe07e6744c058ce6fc4aa187cda96b635539, bfe11d6de1c416cea4f3f0f35f864162063ce3fa, and fbe363c476afe8ec992d3baf682670a4bd1b6ce6 reverted.
First test was done with Dom0 with persistent grant support (Fedora's 3.14.4-200.fc20.x86_64):
1) Partitions:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_pgrants_partitions.png
(same markers mean same bs, we get 860 MB/s here, patches make no difference, result matches expectation)
2) LVM Stripe:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_pgrants_stripe.png
(1715 MB/s, patches make no difference, result matches expectation)
Second test was performed with Dom0 without persistent grants support (Fedora's 3.7.9-205.fc18.x86_64)
1) Partitions:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_nopgrants_partitions.png
(860 MB/sec again, patches worsen a bit overall throughput with 1-3 clients)

Quoted 7 lines

2) LVM Stripe:
http://hadoop.ru/pubfiles/bug1096909/fusion/315_nopgrants_stripe.png
(Here we see the same regression I observed with ramdisks and tmpfs files, unmodified kernel: 1550MB/s, with patches reverted: 1715MB/s).
The only major difference with Felipe's test is that he was using blktap3 with XenServer and I'm using standard blktap2.

Another major difference is that I took older kernels plus the patches instead of taking a 3.15 and reverting the patches.

I'll have a look at your data later in the week. A bit flooded at the moment.

Quoted 3 lines

Signature · 2 lines
-- 
 Vitaly

Vitaly Kuznetsov

2014-06-12 12:00 UTC #22 lore raw