Skip to contentSkip to conversation

SPDK socket abstraction layer

SPDK Walker 3 participants 8 messages Nov 3, 2019 – Nov 7, 2019

lore
Share & tools
lore

Walker

#0 lore raw

On Nov 3, 2019 9:00 AM, Or Gerlitz <gerlitz.or(a)gmail.com> wrote:

Quoted 7 lines · Benjamin

Here's the patch at the top of the series:
https://review.gerrithub.io/c/spdk/spdk/+/471752

Zero copy is getting enabled on the socket and I do see completion notifications, but it's always doing a deferred copy. If there was some description somewhere of what causes the kernel to end up doing a deferred copy instead of page pinning that would be really useful.

the patch you pointed out doesn't change lib/nvmf/tcp.c only the sock code.. can you also refer to the patch that change nvmf

The patches are all in a series with the one I linked at the end. They show in the "relation chain" section of Gerrit. You can click the "Download" button in Gerrit for the patch I linked and it will give you a git-fetch command that grabs the whole series.

#1 lore raw

Hi,

With suggestion of Sasha, we synced the codes to patch (https://review.gerrithub.io/c/spdk/spdk/+/471752). We run tests on two X86-64 servers connected with Mellanox CX-5 100G.

For Perf with 16 QD/4K IO, we found 14% improvement with the zero-copy patch when 1 or 2 cores were used, and 6% improvement when more cores (e.g., 8) were used.

Unfortunately, when we tried to use queue depth more than 16, and bigger IO than 4K, Perf hung or got CQ error. But without the zero-copy patch, 64QD or 64K IO was ok.

Best regards,
Allen

#6 lore raw
In reply to Walker
Quoted 11 lines · Benjamin

On Nov 3, 2019 9:00 AM, Or Gerlitz <gerlitz.or(a)gmail.com> wrote:

Quoted 7 lines · Benjamin

Here's the patch at the top of the series:
https://review.gerrithub.io/c/spdk/spdk/+/471752

Zero copy is getting enabled on the socket and I do see completion notifications, but it's always doing a deferred copy. If there was some description somewhere of what causes the kernel to end up doing a deferred copy instead of page pinning that would be really useful.

Quoted 5 lines

the patch you pointed out doesn't change lib/nvmf/tcp.c only the sock code.. can you also refer to the patch that change nvmf

The patches are all in a series with the one I linked at the end. They show in the "relation chain" section of Gerrit. You can click the "Download" button in Gerrit for the patch I linked and it will give you a git-fetch command that grabs the whole series.

Hi Ben,

yeah, was my bad, I didn't look into the code. Now I see that you have encapsulated the Linux
TX ZC thing inside the socket/posix layer as a follow up change to the async writev concept.

Recently, I worked on the initiator side, basically my approach is different in two aspects:

1. have also the pdu headers on dma memory (this can apply to nvmf as well)

2. since the initiator has transactional communication pattern with the target, I didn't care for zero-copy buffer reclaim notifications from the socket provider,
the target response signal that the buffer may be reused (I don't see how this can
apply to the target side)

Re the approach for 1 && 2 above, please lemmi know if you want to discuss it here or in Gerrit :)

I was working on 19.07 and today rebased it to latest master and pushed that here [1]

I see some issues on the rebased bits when we move from in-capsule to rt2/h2c and
will debug it, just wanted to give you quick access to the bits.

We saw very nice improvements with the 19.07 + TX ZC bits over SmartNIC ARM CPUs and VMA socket accelerator. On this system copy takes at least 25% CPU and we just
eliminated them.

Or.

[1] https://review.gerrithub.io/c/spdk/spdk/+/473280/3

#2 lore raw
In reply to allenz at mellanox.com

Can you outline your system set up so I can reproduce this? On my system currently the network stack is electing to do a deferred copy so the performance is not good. I'd love to reproduce your exact set up.

I'm not surprised there are bugs at higher queue depth. The patch is definitely still a work in progress.

On Nov 4, 2019 9:26 PM, allenz(a)mellanox.com wrote:
Hi,

With suggestion of Sasha, we synced the codes to patch (https://review.gerrithub.io/c/spdk/spdk/+/471752). We run tests on two X86-64 servers connected with Mellanox CX-5 100G.

For Perf with 16 QD/4K IO, we found 14% improvement with the zero-copy patch when 1 or 2 cores were used, and 6% improvement when more cores (e.g., 8) were used.

Unfortunately, when we tried to use queue depth more than 16, and bigger IO than 4K, Perf hung or got CQ error. But without the zero-copy patch, 64QD or 64K IO was ok.

Best regards,
Allen
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

#3 lore raw

Hi,

Our testing environment is:

SW:
CentOS Linux release 8.0.1905 (Core)
Kernel: 4.18.0-80.el8.x86_64
MLNX_OFED_LINUX-4.7-1.0.0.1

HW:
Mellanox Technologies MT27800 Family [ConnectX-5]
CPU: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz

We used null device in target. In both sides (Target, Initiator) we ran the same SPDK version.

Best regards
Sasha

-----Original Message-----
From: Walker, Benjamin <benjamin.walker(a)intel.com>
Sent: Tuesday, November 5, 2019 8:08 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: SPDK socket abstraction layer

Can you outline your system set up so I can reproduce this? On my system currently the network stack is electing to do a deferred copy so the performance is not good. I'd love to reproduce your exact set up.

I'm not surprised there are bugs at higher queue depth. The patch is definitely still a work in progress.

On Nov 4, 2019 9:26 PM, allenz(a)mellanox.com wrote:
Hi,

With suggestion of Sasha, we synced the codes to patch
(https://review.gerrithub.io/c/spdk/spdk/+/471752). We run tests on two X86-64 servers connected with Mellanox CX-5 100G.

For Perf with 16 QD/4K IO, we found 14% improvement with the zero-copy patch when 1 or 2 cores were used, and 6% improvement when more cores (e.g., 8) were used.

Unfortunately, when we tried to use queue depth more than 16, and bigger IO than 4K, Perf hung or got CQ error. But without the zero-copy patch, 64QD or 64K IO was ok.

Best regards,
Allen
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

#7 lore raw
In reply to Or Gerlitz
Quoted 36 lines · Or Gerlitz
Quoted 11 lines · Benjamin

On Nov 3, 2019 9:00 AM, Or Gerlitz <gerlitz.or(a)gmail.com> wrote:

Quoted 7 lines · Benjamin

Here's the patch at the top of the series:
https://review.gerrithub.io/c/spdk/spdk/+/471752

Zero copy is getting enabled on the socket and I do see completion notifications, but it's always doing a deferred copy. If there was some description somewhere of what causes the kernel to end up doing a deferred copy instead of page pinning that would be really useful.

Quoted 5 lines

the patch you pointed out doesn't change lib/nvmf/tcp.c only the sock code.. can you also refer to the patch that change nvmf

The patches are all in a series with the one I linked at the end. They show in the "relation chain" section of Gerrit. You can click the "Download" button in Gerrit for the patch I linked and it will give you a git-fetch command that grabs the whole series.

Hi Ben,

yeah, was my bad, I didn't look into the code. Now I see that you have encapsulated the Linux TX ZC thing inside the socket/posix layer as a follow up change to the async writev concept.

Recently, I worked on the initiator side, basically my approach is different in two aspects:

1. have also the pdu headers on dma memory (this can apply to nvmf as well)

2. since the initiator has transactional communication pattern with the target, I didn't care for zero-copy buffer reclaim notifications from the socket provider, the target response signal that the buffer may be reused (I don't see how this can apply to the target side)

Quoted 7 lines

Re the approach for 1 && 2 above, please lemmi know if you want to discuss it here or in Gerrit :)

I was working on 19.07 and today rebased it to latest master and pushed that here [1]

I see some issues on the rebased bits when we move from in-capsule to rt2/h2c and will debug it, just wanted to give you quick access to the bits.

ok, I had a rebase bug and sorted/fixed it now, pushed the bits [1], will be happy to get
feedback on the approach and the implementation before I go to handle 2-3 small fixmes
I have there (I listed them on the commit change-logs).

[1] https://review.gerrithub.io/c/spdk/spdk/+/473280/5

Quoted 3 lines

We saw very nice improvements with the 19.07 + TX ZC bits over SmartNIC ARM CPUs and VMA socket accelerator. On this system copy takes at least 25% CPU and we just eliminated them.

#4 lore raw
In reply to Walker
Quoted 1 line · Benjamin

On my system currently the network stack is electing to do a deferred copy so the performance is not good.

Ben, can't prove that instantly, but I tend to think that if you use dma buffers for the wire pdu headers,
things would be better for you in that respect (deferred copy), spdk's dma buffers originate (dpdk memory
allocators) from huge pages which afaik on linux are always pinned. Hence the kernel just needs to ref/unref
when you use them with MSG_ZEROCOPY and nothing beyond that. This is the initiator patch [1] but should
be straight forward to apply it for the target.

https://review.gerrithub.io/c/spdk/spdk/+/473278/4

#5 lore raw
Quoted 5 lines · Or Gerlitz
Quoted 2 lines · Benjamin

On my system currently the network stack is electing to do a deferred copy so the performance is not good.

Quoted 9 lines

Ben, can't prove that instantly, but I tend to think that if you use dma buffers for the wire pdu headers,
things would be better for you in that respect (deferred copy), spdk's dma buffers originate (dpdk memory
allocators) from huge pages which afaik on linux are always pinned. Hence the kernel just needs to ref/unref
when you use them with MSG_ZEROCOPY and nothing beyond that. This is the initiator patch [1] but should
be straight forward to apply it for the target.

I agree and I've had similar thoughts - putting all of the data into pre-pinned memory is much more likely to hit a fast path in the pinning logic. But I would have expected it to work even if it wasn't optimal, and it does look like your colleagues at Mellanox got my patch working to some extent. We're just going to try it again using the exact set up they used and confirm that it does function.

In parallel I'm going to change the allocations for the PDUs on the target side to DMA memory like you've done on the initiator.

Quoted 5 lines

https://review.gerrithub.io/c/spdk/spdk/+/473278/4
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org