OVS DPDK DMA-Dev library/Design Discussion

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 7 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely different architectural approach is required:

Many of the DPDK Ethernet PMDs implement a variety of RX and TX packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be treated the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX packet burst functions.

Quoted 7 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to allocate DMA channels and assign them to the specific PMDs that should use them. But the actual use of the DMA channels would move down below the application and into the DPDK PMDs and libraries.

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed & implemented.
1. The DMA configuration and initialization is up to the application (OVS).
2. The VHost library is passed the DMA-dev ID, and its new async rx/tx APIs, and uses the DMA device to accelerate the copy.

Looking forward to talking on the call that just started. Regards, -Harry

Quoted 18 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 7 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely different architectural approach is required:

Many of the DPDK Ethernet PMDs implement a variety of RX and TX packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be treated the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX packet burst functions.

Quoted 7 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to allocate DMA channels and assign them to the specific PMDs that should use them. But the actual use of the DMA channels would move down below the application and into the DPDK PMDs and libraries.

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Hi Morten,

That's *exactly* how this architecture is designed & implemented.
1. The DMA configuration and initialization is up to the application (OVS).
2. The VHost library is passed the DMA-dev ID, and its new async rx/tx APIs, and uses the DMA device to accelerate the copy.

Looking forward to talking on the call that just started. Regards, -Harry

OK, thanks - as I said on the call, I haven't looked at the patches.

Then, I suppose that the TX completions can be handled in the TX function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process TX DMA channel completions. (Effectively, the 2nd pipeline stage.)
2. Pass on the tx_packet_array to the NIC chip descriptors. --> Pass on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

RX_burst(rx_packet_array):
1. Pass on the finished NIC chip RX descriptors to the rx_packet_array. --> Process RX DMA channel completions. (Effectively, the 2nd pipeline stage.)
2. Replenish NIC chip RX descriptors. --> Start RX DMA channel for any waiting packets. (Effectively, the 1nd pipeline stage.)

PMD_init():

(quoted diff) +0 -1

-	Prepare NIC chip RX descriptors. (In other words: Replenish NIC chip RX descriptors. = RX pipeline stage 1.)

PS: Rearranged the email, so we can avoid top posting.

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 7 lines · Morten Brørup

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely different

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be treated the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX packet burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that should use them. But the actual use of the DMA channels would move down below the application and into the DPDK PMDs and libraries.

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed & implemented.
1. The DMA configuration and initialization is up to the application

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new async

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started. Regards, -

Harry

Quoted 5 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the patches.

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process TX

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --> Pass

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view. IOW, the packets will be copied to the Virtio descriptors buffers, but the descriptors will not be made available to the Virtio driver.

Quoted 3 lines

RX_burst(rx_packet_array):
1. Pass on the finished NIC chip RX descriptors to the

rx_packet_array. --> Process RX DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Replenish NIC chip RX descriptors. --> Start RX DMA channel for

any waiting packets. (Effectively, the 1nd pipeline stage.)

Review

Quoted 3 lines

PMD_init():

(quoted diff) +0 -1

-	Prepare NIC chip RX descriptors. (In other words: Replenish NIC

chip RX descriptors. = RX pipeline stage 1.)

Quoted 3 lines

PS: Rearranged the email, so we can avoid top posting.

Quoted 54 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 7 lines · Morten Brørup

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely different

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be treated the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX packet burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that should use them. But the actual use of the DMA channels would move down below the application and into the DPDK PMDs and libraries.

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed & implemented.
1. The DMA configuration and initialization is up to the application

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new async

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started. Regards, -

Harry

Quoted 5 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the patches.

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process TX

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 8 lines

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --> Pass

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view. IOW, the packets will be copied to the Virtio descriptors buffers, but the descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically with an empty array, for completion purposes.

Or some sort of TX_Keepalive() function can be added to the DPDK library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Here is another idea, inspired by a presentation at one of the DPDK Userspace conferences. It may be wishful thinking, though:

Add an additional transaction to each DMA burst; a special transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 9 lines

Quoted 9 lines

the application and into the DPDK PMDs and libraries.

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 2 lines

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is operating in
polling mode. For cases where virtio interrupts are enabled, you still need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

Quoted 89 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 9 lines

Quoted 9 lines

the application and into the DPDK PMDs and libraries.

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 1 line

Quoted 6 lines

Quoted 1 line

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 11 lines

Quoted 1 line

That is something that can work, so long as the receiver is operating in
polling mode. For cases where virtio interrupts are enabled, you still need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario, so let's ignore the DMA TX completion transaction for now - and call it a possible future optimization for specific use cases. So it seems that some form of completion call is unavoidable.

From: Ilya Maximets [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.45

Quoted 55 lines · Morten Brørup

Quoted 55 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 9 lines

Quoted 9 lines

the application and into the DPDK PMDs and libraries.

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 3 lines

Quoted 3 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX

function,

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 4 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 17 lines

Quoted 17 lines

Quoted 4 lines

Quoted 4 lines

pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 1 line

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

I was talking with Maxime after the call today about the same idea. And it looks fairly doable, I would say.

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's

Quoted 1 line

ignore the DMA TX completion transaction for now - and call it a

possible

Quoted 1 line

future optimization for specific use cases. So it seems that some

form of

Quoted 2 lines

completion call is unavoidable.

We could separate the actual kick of the guest with the data transfer. If interrupts are enabled, this means that the guest is not actively polling, i.e. we can allow some extra latency by performing the actual kick from the rx context, or, as Maxime said, if DMA engine can generate
interrupts when the DMA queue is empty, vhost thread may listen to them and kick the guest if needed. This will additionally remove the extra system call from the fast path.

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Wednesday, March 30, 2022 1:45 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; Maxime Coquelin <[email protected]>; Van Haaren, Harry <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; ovs- [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion

Quoted 89 lines · Morten Brørup

Quoted 89 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 8 lines

Quoted 8 lines

the application and into the DPDK PMDs and libraries.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Med venlig hilsen / Kind regards, -Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 2 lines

Here is another idea, inspired by a presentation at one of the DPDK

Quoted 5 lines

Quoted 5 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

I was talking with Maxime after the call today about the same idea. And it looks fairly doable, I would say.

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

That is something that can work, so long as the receiver is operating in polling mode. For cases where virtio interrupts are enabled, you still need to do a write to the eventfd in the kernel in vhost to signal the virtio side. That's not something that can be offloaded to a DMA engine, sadly, so we still need some form of completion call.

Quoted 5 lines

I guess that virtio interrupts is the most widely deployed scenario, so let's ignore the DMA TX completion transaction for now - and call it a possible future optimization for specific use cases. So it seems that some form of completion call is unavoidable.

We could separate the actual kick of the guest with the data transfer. If interrupts are enabled, this means that the guest is not actively polling, i.e. we can allow some extra latency by performing the actual kick from the rx context, or, as Maxime said, if DMA engine can generate interrupts when the DMA queue is empty, vhost thread may listen to them and kick the guest if needed. This will additionally remove the extra system call from the fast path.

Quoted 112 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Wednesday, March 30, 2022 1:45 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; Maxime Coquelin <[email protected]>; Van Haaren, Harry <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; ovs- [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion

Quoted 94 lines · Morten Brørup

Quoted 94 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 8 lines

Quoted 8 lines

the application and into the DPDK PMDs and libraries.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Med venlig hilsen / Kind regards, -Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 2 lines

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 2 lines

I was talking with Maxime after the call today about the same idea. And it looks fairly doable, I would say.

If the idea is making DMA update used ring's index (2B) and packed ring descriptor's flag (2B), yes, it will work functionally. But considering the offloading cost of DMA, it would hurt performance. In addition, the latency of small copy of DMA is much higher than that of CPU. So it will also increase latency.

Quoted 15 lines

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

That is something that can work, so long as the receiver is operating in polling mode. For cases where virtio interrupts are enabled, you still need to do a write to the eventfd in the kernel in vhost to signal the virtio side. That's not something that can be offloaded to a DMA engine, sadly, so we still need some form of completion call.

Quoted 5 lines

I guess that virtio interrupts is the most widely deployed scenario, so let's ignore the DMA TX completion transaction for now - and call it a possible future optimization for specific use cases. So it seems that some form of completion call is unavoidable.

Quoted 7 lines

We could separate the actual kick of the guest with the data transfer. If interrupts are enabled, this means that the guest is not actively polling, i.e. we can allow some extra latency by performing the actual kick from the rx context, or, as Maxime said, if DMA engine can generate interrupts when the DMA queue is empty, vhost thread may listen to them and kick the guest if needed. This will additionally remove the extra system call from the fast path.

Separating kick with data transfer is a very good idea. But it requires a dedicated control plane thread to kick guest after DMA interrupt. Anyway, we can try this optimization in the future.

Thanks,
Jiayu

Quoted 115 lines · Jiayu

Quoted 114 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Wednesday, March 30, 2022 1:45 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; Maxime Coquelin <[email protected]>; Van Haaren, Harry <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; ovs- [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion

Quoted 88 lines · Morten Brørup

Quoted 88 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 8 lines

Quoted 8 lines

the application and into the DPDK PMDs and libraries.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Med venlig hilsen / Kind regards, -Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 1 line

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

I was talking with Maxime after the call today about the same idea. And it looks fairly doable, I would say.

Quoted 4 lines

If the idea is making DMA update used ring's index (2B) and packed ring descriptor's flag (2B), yes, it will work functionally. But considering the offloading cost of DMA, it would hurt performance. In addition, the latency of small copy of DMA is much higher than that of CPU. So it will also increase latency.

I agree writing back descriptors using DMA can be sub-optimal, especially for packed ring where the head desc flags have to be written last.

Are you sure about latency? With current solution, the descriptors write-backs can happen quite some time after the DMA transfers are done, isn't it?

Quoted 23 lines

Quoted 22 lines

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

That is something that can work, so long as the receiver is operating in polling mode. For cases where virtio interrupts are enabled, you still need to do a write to the eventfd in the kernel in vhost to signal the virtio side. That's not something that can be offloaded to a DMA engine, sadly, so we still need some form of completion call.

Quoted 5 lines

I guess that virtio interrupts is the most widely deployed scenario, so let's ignore the DMA TX completion transaction for now - and call it a possible future optimization for specific use cases. So it seems that some form of completion call is unavoidable.

We could separate the actual kick of the guest with the data transfer. If interrupts are enabled, this means that the guest is not actively polling, i.e. we can allow some extra latency by performing the actual kick from the rx context, or, as Maxime said, if DMA engine can generate interrupts when the DMA queue is empty, vhost thread may listen to them and kick the guest if needed. This will additionally remove the extra system call from the fast path.

Quoted 3 lines

Separating kick with data transfer is a very good idea. But it requires a dedicated control plane thread to kick guest after DMA interrupt. Anyway, we can try this optimization in the future.

Yes it requires a dedicated thread, but I don't think this is really an issue. Interrupt mode can be considered as slow-path.

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Wednesday, March 30, 2022 5:25 PM
To: Hu, Jiayu <[email protected]>; Ilya Maximets <[email protected]>; Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Van Haaren, Harry <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Ferriter, Cian <[email protected]>; [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion

Quoted 8 lines · Jiayu

Quoted 8 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Wednesday, March 30, 2022 1:45 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; Maxime Coquelin

<[email protected]>;

Quoted 107 lines

Quoted 107 lines

Van Haaren, Harry <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; ovs- [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion

Quoted 97 lines · Morten Brørup

Quoted 97 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 6 lines

Quoted 6 lines

architectural approach is required:

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 8 lines

Quoted 8 lines

the application and into the DPDK PMDs and libraries.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Med venlig hilsen / Kind regards, -Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 4 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 2 lines

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 3 lines

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 7 lines

Quoted 2 lines

I was talking with Maxime after the call today about the same idea. And it looks fairly doable, I would say.

If the idea is making DMA update used ring's index (2B) and packed ring descriptor's flag (2B), yes, it will work functionally. But considering the offloading cost of DMA, it would hurt performance. In addition, the latency of small copy of DMA is much higher than that of CPU.

So it will also increase latency.

I agree writing back descriptors using DMA can be sub-optimal, especially for packed ring where the head desc flags have to be written last.

Are you sure about latency? With current solution, the descriptors write- backs can happen quite some time after the DMA transfers are done, isn't it?

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 3 lines

That is something that can work, so long as the receiver is operating in polling mode. For cases where virtio interrupts are enabled, you still need to do a write to the eventfd in the kernel in vhost to signal the virtio side. That's not something that can be offloaded to a DMA engine, sadly, so we still need some form of

completion call.

Quoted 9 lines

Quoted 9 lines

Quoted 6 lines

I guess that virtio interrupts is the most widely deployed scenario, so let's ignore the DMA TX completion transaction for now - and call it a possible future optimization for specific use cases. So it seems that some form of completion call is unavoidable.

We could separate the actual kick of the guest with the data transfer. If interrupts are enabled, this means that the guest is not actively polling,

i.e.

Quoted 6 lines

Quoted 5 lines

we can allow some extra latency by performing the actual kick from the rx context, or, as Maxime said, if DMA engine can generate interrupts when the DMA queue is empty, vhost thread may listen to them and kick the guest if needed. This will additionally remove the extra system call from the fast path.

Quoted 3 lines

Separating kick with data transfer is a very good idea. But it requires a dedicated control plane thread to kick guest after DMA interrupt. Anyway, we can try this optimization in the future.

Yes it requires a dedicated thread, but I don't think this is really an issue. Interrupt mode can be considered as slow-path.

Quoted 4 lines

Thanks,
Jiayu

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM
To: Richardson, Bruce <[email protected]>

Cc: Maxime Coquelin <[email protected]>; Van Haaren, Harry <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; ovs- [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 83 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 5 lines

Quoted 5 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX and TX

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 2 lines

Quoted 2 lines

the same way. So I suggest that PMDs copying packet contents, e.g. memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 6 lines

Quoted 6 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job to

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move down

below

Quoted 9 lines

Quoted 9 lines

the application and into the DPDK PMDs and libraries.

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its new

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the TX

function, and the RX completions can be handled in the RX function, just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. --> Process

TX

Quoted 8 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip descriptors. --

Pass

Quoted 5 lines

on the tx_packet_array to the TX DMA channel. (Effectively, the 1st pipeline stage.)

The problem is Tx function might not be called again, so enqueued packets in 2. may never be completed from a Virtio point of view.

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers, but

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst() periodically

with an empty array, for completion purposes.

Quoted 4 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.

Quoted 8 lines

Quoted 1 line

Here is another idea, inspired by a presentation at one of the DPDK

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

Quoted 9 lines

That is something that can work, so long as the receiver is operating in
polling mode. For cases where virtio interrupts are enabled, you still need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario, so let's ignore the DMA TX completion transaction for now - and call it a possible future optimization for specific use cases. So it seems that some form of completion call is unavoidable.

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.46

Quoted 24 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM

Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 3 lines

Quoted 3 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX

and TX

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 1 line

Quoted 1 line

the same way. So I suggest that PMDs copying packet contents,

e.g.

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

Quoted 1 line

memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 5 lines

Quoted 5 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job

to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move

down

Quoted 7 lines

Quoted 7 lines

below

Quoted 6 lines

Quoted 6 lines

the application and into the DPDK PMDs and libraries.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its

new

Quoted 4 lines

Quoted 4 lines

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Quoted 8 lines

Quoted 8 lines

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the

TX

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

function, and the RX completions can be handled in the RX

function,

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. -->

Process

Quoted 3 lines

Quoted 3 lines

TX

Quoted 2 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip

descriptors. --

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

pipeline stage.)

The problem is Tx function might not be called again, so

enqueued

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

packets in 2. may never be completed from a Virtio point of

view.

Quoted 2 lines

Quoted 2 lines

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers,

but

Quoted 4 lines

Quoted 4 lines

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst()

periodically

Quoted 1 line

Quoted 1 line

with an empty array, for completion purposes.

This is what the "defer work" does at the OVS thread-level, but instead of
"brute-forcing" and *always* making the call, the defer work concept tracks
*when* there is outstanding work (DMA copies) to be completed ("deferred work")
and calls the generic completion function at that point.

So "defer work" is generic infrastructure at the OVS thread level to handle
work that needs to be done "later", e.g. DMA completion handling.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple

DMA

Quoted 2 lines

Quoted 2 lines

channels, if convenient - and if possible without locking or other weird complexity.

That's exactly how it is done, the VHost library has a new API added, which allows
for handling completions. And in the "Netdev layer" (~OVS ethdev abstraction)
we add a function to allow the OVS thread to do those completions in a new
Netdev-abstraction API called "async_process" where the completions can be checked.

The only method to abstract them is to "hide" them somewhere that will always be
polled, e.g. an ethdev port's RX function. Both V3 and V4 approaches use this method.
This allows "completions" to be transparent to the app, at the tradeoff to having bad
separation of concerns as Rx and Tx are now tied-together.

The point is, the Application layer must *somehow * handle of completions.
So fundamentally there are 2 options for the Application level:

A) Make the application periodically call a "handle completions" function
A1) Defer work, call when needed, and track "needed" at app
layer, and calling into vhost txq complete as required.
Elegant in that "no work" means "no cycles spent" on
checking DMA completions.
A2) Brute-force-always-call, and pay some overhead when not
required.
Cycle-cost in "no work" scenarios. Depending on # of
vhost queues, this adds up as polling required *per vhost txq*.
Also note that "checking DMA completions" means taking a
virtq-lock, so this "brute-force" can needlessly increase x-thread contention!

B) Hide completions and live with the complexity/architectural sacrifice of mixed-RxTx.
Various downsides here in my opinion, see the slide deck
presented earlier today for a summary.

In my opinion, A1 is the most elegant solution, as it has a clean separation of concerns, does not cause
avoidable contention on virtq locks, and spends no cycles when there is no completion work to do.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Here is another idea, inspired by a presentation at one of the

DPDK

Quoted 8 lines

Quoted 8 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's ignore

Quoted 1 line

the DMA TX completion transaction for now - and call it a possible

future

Quoted 1 line

optimization for specific use cases. So it seems that some form of

completion call

Quoted 1 line

is unavoidable.

Agree to leave this aside, there is in theory a potential optimization, but
unlikely to be of large value.

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 8:59 PM
To: Van Haaren, Harry <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 144 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.46

Quoted 24 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM

Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 3 lines

Quoted 3 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX

and TX

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 1 line

Quoted 1 line

the same way. So I suggest that PMDs copying packet contents,

e.g.

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

Quoted 1 line

memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 5 lines

Quoted 5 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job

to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move

down

Quoted 7 lines

Quoted 7 lines

below

Quoted 6 lines

Quoted 6 lines

the application and into the DPDK PMDs and libraries.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its

new

Quoted 4 lines

Quoted 4 lines

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Quoted 8 lines

Quoted 8 lines

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the

TX

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

function, and the RX completions can be handled in the RX

function,

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. -->

Process

Quoted 3 lines

Quoted 3 lines

TX

Quoted 2 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip

descriptors. --

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

pipeline stage.)

The problem is Tx function might not be called again, so

enqueued

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

packets in 2. may never be completed from a Virtio point of

view.

Quoted 2 lines

Quoted 2 lines

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers,

but

Quoted 4 lines

Quoted 4 lines

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst()

periodically

Quoted 1 line

Quoted 1 line

with an empty array, for completion purposes.

This is what the "defer work" does at the OVS thread-level, but instead of
"brute-forcing" and *always* making the call, the defer work concept tracks
*when* there is outstanding work (DMA copies) to be completed ("deferred work")
and calls the generic completion function at that point.

So "defer work" is generic infrastructure at the OVS thread level to handle
work that needs to be done "later", e.g. DMA completion handling.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple

DMA

Quoted 2 lines

Quoted 2 lines

channels, if convenient - and if possible without locking or other weird complexity.

That's exactly how it is done, the VHost library has a new API added, which allows
for handling completions. And in the "Netdev layer" (~OVS ethdev abstraction)
we add a function to allow the OVS thread to do those completions in a new
Netdev-abstraction API called "async_process" where the completions can be checked.

The only method to abstract them is to "hide" them somewhere that will always be
polled, e.g. an ethdev port's RX function. Both V3 and V4 approaches use this method.
This allows "completions" to be transparent to the app, at the tradeoff to having bad
separation of concerns as Rx and Tx are now tied-together.

The point is, the Application layer must *somehow * handle of completions.
So fundamentally there are 2 options for the Application level:

Quoted 13 lines

A) Make the application periodically call a "handle completions" function
A1) Defer work, call when needed, and track "needed" at app
layer, and calling into vhost txq complete as required.
Elegant in that "no work" means "no cycles spent" on
checking DMA completions.
A2) Brute-force-always-call, and pay some overhead when not
required.
Cycle-cost in "no work" scenarios. Depending on # of
vhost queues, this adds up as polling required *per vhost txq*.
Also note that "checking DMA completions" means taking a
virtq-lock, so this "brute-force" can needlessly increase x-thread contention!

A side note: I don't see why locking is required to test for DMA completions. rte_dma_vchan_status() is lockless, e.g.:
https://elixir.bootlin.com/dpdk/latest/source/drivers/dma/ioat/ioat_dmadev.c#L56
0

Quoted 5 lines

B) Hide completions and live with the complexity/architectural sacrifice of mixed-RxTx.
Various downsides here in my opinion, see the slide deck
presented earlier today for a summary.

Quoted 5 lines

In my opinion, A1 is the most elegant solution, as it has a clean separation of concerns, does not cause
avoidable contention on virtq locks, and spends no cycles when there is no completion work to do.

Thank you for elaborating, Harry.

I strongly oppose against hiding any part of TX processing in an RX function. It is just wrong in so many ways!

I agree that A1 is the most elegant solution. And being the most elegant solution, it is probably also the most future proof solution. :-)

I would also like to stress that DMA completion handling belongs in the DPDK library, not in the application. And yes, the application will be required to call some "handle DMA completions" function in the DPDK library. But since the application already knows that it uses DMA, the application should also know that it needs to call this extra function - so I consider this requirement perfectly acceptable.

I prefer if the DPDK vhost library can hide its inner workings from the application, and just expose the additional "handle completions" function. This also means that the inner workings can be implemented as "defer work", or by some other algorithm. And it can be tweaked and optimized later.

Thinking about the long term perspective, this design pattern is common for both the vhost library and other DPDK libraries that could benefit from DMA (e.g. vmxnet3 and pcap PMDs), so it could be abstracted into the DMA library or a separate library. But for now, we should focus on the vhost use case, and just keep the long term roadmap for using DMA in mind.

Rephrasing what I said on the conference call: This vhost design will become the common design pattern for using DMA in DPDK libraries. If we get it wrong, we are stuck with it.

Quoted 24 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Here is another idea, inspired by a presentation at one of the

DPDK

Quoted 8 lines

Quoted 8 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's ignore

Quoted 1 line

the DMA TX completion transaction for now - and call it a possible

Quoted 9 lines

future

Quoted 1 line

optimization for specific use cases. So it seems that some form of

completion call

Quoted 1 line

is unavoidable.

Agree to leave this aside, there is in theory a potential optimization, but
unlikely to be of large value.

One more thing: When using DMA to pass on packets into a guest, there could be a delay from the DMA completes until the guest is signaled. Is there any CPU cache hotness regarding the guest's access to the packet data to consider here? I.e. if we wait signaling the guest, the packet data may get cold.

-----Original Message-----
From: Van Haaren, Harry
Sent: Wednesday, March 30, 2022 10:02 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 9 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 8:59 PM
To: Van Haaren, Harry <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara,

John

Quoted 162 lines

<[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 144 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.46

Quoted 24 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM

Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 3 lines

Quoted 3 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX

and TX

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 1 line

Quoted 1 line

the same way. So I suggest that PMDs copying packet contents,

e.g.

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

Quoted 1 line

memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 5 lines

Quoted 5 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job

to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move

down

Quoted 7 lines

Quoted 7 lines

below

Quoted 6 lines

Quoted 6 lines

the application and into the DPDK PMDs and libraries.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its

new

Quoted 4 lines

Quoted 4 lines

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Quoted 8 lines

Quoted 8 lines

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the

TX

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

function, and the RX completions can be handled in the RX

function,

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. -->

Process

Quoted 3 lines

Quoted 3 lines

TX

Quoted 2 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip

descriptors. --

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

pipeline stage.)

The problem is Tx function might not be called again, so

enqueued

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

packets in 2. may never be completed from a Virtio point of

view.

Quoted 2 lines

Quoted 2 lines

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers,

but

Quoted 4 lines

Quoted 4 lines

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst()

periodically

Quoted 1 line

Quoted 1 line

with an empty array, for completion purposes.

This is what the "defer work" does at the OVS thread-level, but instead of
"brute-forcing" and *always* making the call, the defer work concept tracks
*when* there is outstanding work (DMA copies) to be completed ("deferred work")
and calls the generic completion function at that point.

So "defer work" is generic infrastructure at the OVS thread level to handle
work that needs to be done "later", e.g. DMA completion handling.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple

DMA

Quoted 2 lines

Quoted 2 lines

channels, if convenient - and if possible without locking or other weird complexity.

That's exactly how it is done, the VHost library has a new API added, which allows
for handling completions. And in the "Netdev layer" (~OVS ethdev abstraction)
we add a function to allow the OVS thread to do those completions in a new
Netdev-abstraction API called "async_process" where the completions can be checked.

The only method to abstract them is to "hide" them somewhere that will always be
polled, e.g. an ethdev port's RX function. Both V3 and V4 approaches use this method.
This allows "completions" to be transparent to the app, at the tradeoff to having bad
separation of concerns as Rx and Tx are now tied-together.

The point is, the Application layer must *somehow * handle of completions.
So fundamentally there are 2 options for the Application level:

Quoted 13 lines

A) Make the application periodically call a "handle completions" function
A1) Defer work, call when needed, and track "needed" at app
layer, and calling into vhost txq complete as required.
Elegant in that "no work" means "no cycles spent" on
checking DMA completions.
A2) Brute-force-always-call, and pay some overhead when not
required.
Cycle-cost in "no work" scenarios. Depending on # of
vhost queues, this adds up as polling required *per vhost txq*.
Also note that "checking DMA completions" means taking a
virtq-lock, so this "brute-force" can needlessly increase x-thread contention!

Quoted 3 lines

A side note: I don't see why locking is required to test for DMA completions. rte_dma_vchan_status() is lockless, e.g.:

https://elixir.bootlin.com/dpdk/latest/source/drivers/dma/ioat/ioat_dmadev.c#L
56

Quoted 1 line

0

Correct, DMA-dev is "ethdev like"; each DMA-id can be used in a lockfree manner from a single thread.

The locks I refer to are at the OVS-netdev level, as virtq's are shared across OVS's dataplane threads.
So the "M to N" comes from M dataplane threads to N virtqs, hence requiring some locking.

Quoted 5 lines

Quoted 5 lines

B) Hide completions and live with the complexity/architectural sacrifice of mixed-RxTx.
Various downsides here in my opinion, see the slide deck
presented earlier today for a summary.

Quoted 7 lines

Quoted 5 lines

In my opinion, A1 is the most elegant solution, as it has a clean separation of concerns, does not cause
avoidable contention on virtq locks, and spends no cycles when there is no completion work to do.

Thank you for elaborating, Harry.

Thanks for part-taking in the discussion & providing your insight!

Quoted 1 line

I strongly oppose against hiding any part of TX processing in an RX function. It

is just

Quoted 3 lines

wrong in so many ways!

I agree that A1 is the most elegant solution. And being the most elegant

solution, it

Quoted 1 line

is probably also the most future proof solution. :-)

I think so too, yes.

Quoted 2 lines

I would also like to stress that DMA completion handling belongs in the DPDK library, not in the application. And yes, the application will be required to call

some

Quoted 1 line

"handle DMA completions" function in the DPDK library. But since the

application

Quoted 1 line

already knows that it uses DMA, the application should also know that it needs

to

Quoted 1 line

call this extra function - so I consider this requirement perfectly acceptable.

Agree here.

Quoted 1 line

I prefer if the DPDK vhost library can hide its inner workings from the

application,

Quoted 1 line

and just expose the additional "handle completions" function. This also means

that

Quoted 2 lines

the inner workings can be implemented as "defer work", or by some other algorithm. And it can be tweaked and optimized later.

Yes, the choice in how to call the handle_completions function is Application layer.
For OVS we designed Defer Work, V3 and V4. But it is an App level choice, and every
application is free to choose its own method.

Quoted 1 line

Thinking about the long term perspective, this design pattern is common for

both

Quoted 3 lines

the vhost library and other DPDK libraries that could benefit from DMA (e.g. vmxnet3 and pcap PMDs), so it could be abstracted into the DMA library or a separate library. But for now, we should focus on the vhost use case, and just

keep

Quoted 1 line

the long term roadmap for using DMA in mind.

Totally agree to keep long term roadmap in mind; but I'm not sure we can refactor
logic out of vhost. When DMA-completions arrive, the virtQ needs to be updated;
this causes a tight coupling between the DMA completion count, and the vhost library.

As Ilya raised on the call yesterday, there is an "in_order" requirement in the vhost
library, that per virtq the packets are presented to the guest "in order" of enqueue.
(To be clear, *not* order of DMA-completion! As Jiayu mentioned, the Vhost library
handles this today by re-ordering the DMA completions.)

Quoted 1 line

Rephrasing what I said on the conference call: This vhost design will become

the

Quoted 1 line

common design pattern for using DMA in DPDK libraries. If we get it wrong, we

are

Quoted 1 line

stuck with it.

Agree, and if we get it right, then we're stuck with it too! :)

Quoted 27 lines

Quoted 27 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Here is another idea, inspired by a presentation at one of the

DPDK

Quoted 8 lines

Quoted 8 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's ignore

Quoted 1 line

the DMA TX completion transaction for now - and call it a possible

future

Quoted 1 line

optimization for specific use cases. So it seems that some form of

completion call

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

is unavoidable.

Agree to leave this aside, there is in theory a potential optimization, but
unlikely to be of large value.

One more thing: When using DMA to pass on packets into a guest, there could

be a

Quoted 1 line

delay from the DMA completes until the guest is signaled. Is there any CPU

cache

Quoted 1 line

hotness regarding the guest's access to the packet data to consider here? I.e. if

we

Quoted 1 line

wait signaling the guest, the packet data may get cold.

Interesting question; we can likely spawn a new thread around this topic! In short, it depends on how/where the DMA hardware writes the copy.

With technologies like DDIO, the "dest" part of the copy will be in LLC. The core reading the
dest data will benefit from the LLC locality (instead of snooping it from a remote core's L1/L2).

Delays in notifying the guest could result in LLC capacity eviction, yes. The application layer decides how often/promptly to check for completions, and notify the guest of them. Calling the function more often will result in less delay in that portion of the pipeline.

Overall, there are caching benefits with DMA acceleration, and the application can control
the latency introduced between dma-completion done in HW, and Guest vring update.

Hi OVS & DPDK, Maintainers & Community,

Top posting overview of discussion as replies to thread become slower:
perhaps it is a good time to review and plan for next steps?

From my perspective, it those most vocal in the thread seem to be in favour of the clean
rx/tx split ("defer work"), with the tradeoff that the application must be aware of handling the async DMA completions. If there are any concerns opposing upstreaming of this method, please indicate this promptly, and we can continue technical discussions here now.

In absence of continued technical discussion here, I suggest Sunil and Ian collaborate on getting the OVS Defer-work approach, and DPDK VHost Async patchsets available on GitHub for easier consumption and future development (as suggested in slides presented on last call).

Regards, -Harry

No inline-replies below; message just for context.

Quoted 342 lines

-----Original Message-----
From: Van Haaren, Harry
Sent: Wednesday, March 30, 2022 10:02 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 9 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 8:59 PM
To: Van Haaren, Harry <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara,

John

Quoted 162 lines

<[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 144 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.46

Quoted 24 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM

Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 3 lines

Quoted 3 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX

and TX

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 1 line

Quoted 1 line

the same way. So I suggest that PMDs copying packet contents,

e.g.

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

Quoted 1 line

memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 5 lines

Quoted 5 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job

to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move

down

Quoted 7 lines

Quoted 7 lines

below

Quoted 6 lines

Quoted 6 lines

the application and into the DPDK PMDs and libraries.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1. The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2. The VHost library is passed the DMA-dev ID, and its

new

Quoted 4 lines

Quoted 4 lines

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Quoted 8 lines

Quoted 8 lines

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the

TX

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

function, and the RX completions can be handled in the RX

function,

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1. Clean up descriptors processed by the NIC chip. -->

Process

Quoted 3 lines

Quoted 3 lines

TX

Quoted 2 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2. Pass on the tx_packet_array to the NIC chip

descriptors. --

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

pipeline stage.)

The problem is Tx function might not be called again, so

enqueued

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

packets in 2. may never be completed from a Virtio point of

view.

Quoted 2 lines

Quoted 2 lines

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers,

but

Quoted 4 lines

Quoted 4 lines

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst()

periodically

Quoted 1 line

Quoted 1 line

with an empty array, for completion purposes.

This is what the "defer work" does at the OVS thread-level, but instead of
"brute-forcing" and *always* making the call, the defer work concept tracks
*when* there is outstanding work (DMA copies) to be completed ("deferred work")
and calls the generic completion function at that point.

So "defer work" is generic infrastructure at the OVS thread level to handle
work that needs to be done "later", e.g. DMA completion handling.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple

DMA

Quoted 2 lines

Quoted 2 lines

channels, if convenient - and if possible without locking or other weird complexity.

That's exactly how it is done, the VHost library has a new API added, which allows
for handling completions. And in the "Netdev layer" (~OVS ethdev abstraction)
we add a function to allow the OVS thread to do those completions in a new
Netdev-abstraction API called "async_process" where the completions can be checked.

The only method to abstract them is to "hide" them somewhere that will always be
polled, e.g. an ethdev port's RX function. Both V3 and V4 approaches use this method.
This allows "completions" to be transparent to the app, at the tradeoff to having bad
separation of concerns as Rx and Tx are now tied-together.

The point is, the Application layer must *somehow * handle of completions.
So fundamentally there are 2 options for the Application level:

Quoted 13 lines

A) Make the application periodically call a "handle completions" function
A1) Defer work, call when needed, and track "needed" at app
layer, and calling into vhost txq complete as required.
Elegant in that "no work" means "no cycles spent" on
checking DMA completions.
A2) Brute-force-always-call, and pay some overhead when not
required.
Cycle-cost in "no work" scenarios. Depending on # of
vhost queues, this adds up as polling required *per vhost txq*.
Also note that "checking DMA completions" means taking a
virtq-lock, so this "brute-force" can needlessly increase x-thread contention!

Quoted 3 lines

A side note: I don't see why locking is required to test for DMA completions. rte_dma_vchan_status() is lockless, e.g.:

https://elixir.bootlin.com/dpdk/latest/source/drivers/dma/ioat/ioat_dmadev.c#L
56

Quoted 1 line

0

Correct, DMA-dev is "ethdev like"; each DMA-id can be used in a lockfree manner from a single thread.

The locks I refer to are at the OVS-netdev level, as virtq's are shared across OVS's dataplane threads.
So the "M to N" comes from M dataplane threads to N virtqs, hence requiring some locking.

Quoted 5 lines

Quoted 5 lines

B) Hide completions and live with the complexity/architectural sacrifice of mixed-RxTx.
Various downsides here in my opinion, see the slide deck
presented earlier today for a summary.

Quoted 7 lines

Quoted 5 lines

In my opinion, A1 is the most elegant solution, as it has a clean separation of concerns, does not cause
avoidable contention on virtq locks, and spends no cycles when there is no completion work to do.

Thank you for elaborating, Harry.

Thanks for part-taking in the discussion & providing your insight!

Quoted 1 line

I strongly oppose against hiding any part of TX processing in an RX function. It

is just

Quoted 3 lines

wrong in so many ways!

I agree that A1 is the most elegant solution. And being the most elegant

solution, it

Quoted 1 line

is probably also the most future proof solution. :-)

I think so too, yes.

Quoted 2 lines

I would also like to stress that DMA completion handling belongs in the DPDK library, not in the application. And yes, the application will be required to call

some

Quoted 1 line

"handle DMA completions" function in the DPDK library. But since the

application

Quoted 1 line

already knows that it uses DMA, the application should also know that it needs

to

Quoted 1 line

call this extra function - so I consider this requirement perfectly acceptable.

Agree here.

Quoted 1 line

I prefer if the DPDK vhost library can hide its inner workings from the

application,

Quoted 1 line

and just expose the additional "handle completions" function. This also means

that

Quoted 2 lines

the inner workings can be implemented as "defer work", or by some other algorithm. And it can be tweaked and optimized later.

Yes, the choice in how to call the handle_completions function is Application layer.
For OVS we designed Defer Work, V3 and V4. But it is an App level choice, and every
application is free to choose its own method.

Quoted 1 line

Thinking about the long term perspective, this design pattern is common for

both

Quoted 3 lines

the vhost library and other DPDK libraries that could benefit from DMA (e.g. vmxnet3 and pcap PMDs), so it could be abstracted into the DMA library or a separate library. But for now, we should focus on the vhost use case, and just

keep

Quoted 1 line

the long term roadmap for using DMA in mind.

Totally agree to keep long term roadmap in mind; but I'm not sure we can refactor
logic out of vhost. When DMA-completions arrive, the virtQ needs to be updated;
this causes a tight coupling between the DMA completion count, and the vhost library.

As Ilya raised on the call yesterday, there is an "in_order" requirement in the vhost
library, that per virtq the packets are presented to the guest "in order" of enqueue.
(To be clear, *not* order of DMA-completion! As Jiayu mentioned, the Vhost library
handles this today by re-ordering the DMA completions.)

Quoted 1 line

Rephrasing what I said on the conference call: This vhost design will become

the

Quoted 1 line

common design pattern for using DMA in DPDK libraries. If we get it wrong, we

are

Quoted 1 line

stuck with it.

Agree, and if we get it right, then we're stuck with it too! :)

Quoted 27 lines

Quoted 27 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Here is another idea, inspired by a presentation at one of the

DPDK

Quoted 8 lines

Quoted 8 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's ignore

Quoted 1 line

the DMA TX completion transaction for now - and call it a possible

future

Quoted 1 line

optimization for specific use cases. So it seems that some form of

completion call

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

is unavoidable.

Agree to leave this aside, there is in theory a potential optimization, but
unlikely to be of large value.

One more thing: When using DMA to pass on packets into a guest, there could

be a

Quoted 1 line

delay from the DMA completes until the guest is signaled. Is there any CPU

cache

Quoted 1 line

hotness regarding the guest's access to the packet data to consider here? I.e. if

we

Quoted 1 line

wait signaling the guest, the packet data may get cold.

Interesting question; we can likely spawn a new thread around this topic! In short, it depends on how/where the DMA hardware writes the copy.

With technologies like DDIO, the "dest" part of the copy will be in LLC. The core reading the
dest data will benefit from the LLC locality (instead of snooping it from a remote core's L1/L2).

Delays in notifying the guest could result in LLC capacity eviction, yes. The application layer decides how often/promptly to check for completions, and notify the guest of them. Calling the function more often will result in less delay in that portion of the pipeline.

Overall, there are caching benefits with DMA acceleration, and the application can control
the latency introduced between dma-completion done in HW, and Guest vring update.

Hi Harry,

Quoted 5 lines · Harry

Hi OVS & DPDK, Maintainers & Community,

Top posting overview of discussion as replies to thread become slower:
perhaps it is a good time to review and plan for next steps?

Quoted 4 lines

 From my perspective, it those most vocal in the thread seem to be in favour of the clean rx/tx split ("defer work"), with the tradeoff that the application must be aware of handling the async DMA completions. If there are any concerns opposing upstreaming of this method, please indicate this promptly, and we can continue technical discussions here now.

Wasn't there some discussions about handling the Virtio completions with the DMA engine? With that, we wouldn't need the deferral of work.

Thanks,
Maxime

Quoted 351 lines

In absence of continued technical discussion here, I suggest Sunil and Ian collaborate on getting the OVS Defer-work approach, and DPDK VHost Async patchsets available on GitHub for easier consumption and future development (as suggested in slides presented on last call).

Regards, -Harry

No inline-replies below; message just for context.

Quoted 342 lines

-----Original Message-----
From: Van Haaren, Harry
Sent: Wednesday, March 30, 2022 10:02 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 9 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 8:59 PM
To: Van Haaren, Harry <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya Maximets <[email protected]>; [email protected]; [email protected]; Mcnamara,

John

Quoted 162 lines

<[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 144 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.46

Quoted 24 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM

Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 3 lines

Quoted 3 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX

and TX

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 1 line

Quoted 1 line

the same way. So I suggest that PMDs copying packet contents,

e.g.

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

Quoted 1 line

memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 5 lines

Quoted 5 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job

to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move

down

Quoted 7 lines

Quoted 7 lines

below

Quoted 6 lines

Quoted 6 lines

the application and into the DPDK PMDs and libraries.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1.    The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2.    The VHost library is passed the DMA-dev ID, and its

new

Quoted 4 lines

Quoted 4 lines

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Quoted 8 lines

Quoted 8 lines

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the

TX

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

function, and the RX completions can be handled in the RX

function,

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1.    Clean up descriptors processed by the NIC chip. -->

Process

Quoted 3 lines

Quoted 3 lines

TX

Quoted 2 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2.    Pass on the tx_packet_array to the NIC chip

descriptors. --

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

pipeline stage.)

The problem is Tx function might not be called again, so

enqueued

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

packets in 2. may never be completed from a Virtio point of

view.

Quoted 2 lines

Quoted 2 lines

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers,

but

Quoted 4 lines

Quoted 4 lines

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst()

periodically

Quoted 1 line

Quoted 1 line

with an empty array, for completion purposes.

This is what the "defer work" does at the OVS thread-level, but instead of
"brute-forcing" and *always* making the call, the defer work concept tracks
*when* there is outstanding work (DMA copies) to be completed ("deferred work")
and calls the generic completion function at that point.

So "defer work" is generic infrastructure at the OVS thread level to handle
work that needs to be done "later", e.g. DMA completion handling.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple

DMA

Quoted 2 lines

Quoted 2 lines

channels, if convenient - and if possible without locking or other weird complexity.

That's exactly how it is done, the VHost library has a new API added, which allows
for handling completions. And in the "Netdev layer" (~OVS ethdev abstraction)
we add a function to allow the OVS thread to do those completions in a new
Netdev-abstraction API called "async_process" where the completions can be checked.

The only method to abstract them is to "hide" them somewhere that will always be
polled, e.g. an ethdev port's RX function.  Both V3 and V4 approaches use this method.
This allows "completions" to be transparent to the app, at the tradeoff to having bad
separation  of concerns as Rx and Tx are now tied-together.

The point is, the Application layer must *somehow * handle of completions.
So fundamentally there are 2 options for the Application level:

Quoted 13 lines

A) Make the application periodically call a "handle completions" function
    A1) Defer work, call when needed, and track "needed" at app layer, and calling into vhost txq complete as required. Elegant in that "no work" means "no cycles spent" on checking DMA completions.
    A2) Brute-force-always-call, and pay some overhead when not required.
            Cycle-cost in "no work" scenarios. Depending on # of vhost queues, this adds up as polling required *per vhost txq*. Also note that "checking DMA completions" means taking a virtq-lock, so this "brute-force" can needlessly increase x-thread contention!

Quoted 3 lines

A side note: I don't see why locking is required to test for DMA completions. rte_dma_vchan_status() is lockless, e.g.:

https://elixir.bootlin.com/dpdk/latest/source/drivers/dma/ioat/ioat_dmadev.c#L
56

Quoted 1 line

0

Correct, DMA-dev is "ethdev like"; each DMA-id can be used in a lockfree manner from a single thread.

The locks I refer to are at the OVS-netdev level, as virtq's are shared across OVS's dataplane threads.
So the "M to N" comes from M dataplane threads to N virtqs, hence requiring some locking.

Quoted 5 lines

Quoted 5 lines

B) Hide completions and live with the complexity/architectural sacrifice of mixed-RxTx.
    Various downsides here in my opinion, see the slide deck presented earlier today for a summary.

Quoted 7 lines

Quoted 5 lines

In my opinion, A1 is the most elegant solution, as it has a clean separation of concerns, does not  cause
avoidable contention on virtq locks, and spends no cycles when there is no completion work to do.

Thank you for elaborating, Harry.

Thanks for part-taking in the discussion & providing your insight!

Quoted 1 line

I strongly oppose against hiding any part of TX processing in an RX function. It

is just

Quoted 3 lines

wrong in so many ways!

I agree that A1 is the most elegant solution. And being the most elegant

solution, it

Quoted 1 line

is probably also the most future proof solution. :-)

I think so too, yes.

Quoted 2 lines

I would also like to stress that DMA completion handling belongs in the DPDK library, not in the application. And yes, the application will be required to call

some

Quoted 1 line

"handle DMA completions" function in the DPDK library. But since the

application

Quoted 1 line

already knows that it uses DMA, the application should also know that it needs

to

Quoted 1 line

call this extra function - so I consider this requirement perfectly acceptable.

Agree here.

Quoted 1 line

I prefer if the DPDK vhost library can hide its inner workings from the

application,

Quoted 1 line

and just expose the additional "handle completions" function. This also means

that

Quoted 2 lines

the inner workings can be implemented as "defer work", or by some other algorithm. And it can be tweaked and optimized later.

Yes, the choice in how to call the handle_completions function is Application layer.
For OVS we designed Defer Work, V3 and V4. But it is an App level choice, and every
application is free to choose its own method.

Quoted 1 line

Thinking about the long term perspective, this design pattern is common for

both

Quoted 3 lines

the vhost library and other DPDK libraries that could benefit from DMA (e.g. vmxnet3 and pcap PMDs), so it could be abstracted into the DMA library or a separate library. But for now, we should focus on the vhost use case, and just

keep

Quoted 1 line

the long term roadmap for using DMA in mind.

Totally agree to keep long term roadmap in mind; but I'm not sure we can refactor
logic out of vhost. When DMA-completions arrive, the virtQ needs to be updated;
this causes a tight coupling between the DMA completion count, and the vhost library.

As Ilya raised on the call yesterday, there is an "in_order" requirement in the vhost
library, that per virtq the packets are presented to the guest "in order" of enqueue.
(To be clear, *not* order of DMA-completion! As Jiayu mentioned, the Vhost library
handles this today by re-ordering the DMA completions.)

Quoted 1 line

Rephrasing what I said on the conference call: This vhost design will become

the

Quoted 1 line

common design pattern for using DMA in DPDK libraries. If we get it wrong, we

are

Quoted 1 line

stuck with it.

Agree, and if we get it right, then we're stuck with it too! :)

Quoted 27 lines

Quoted 27 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Here is another idea, inspired by a presentation at one of the

DPDK

Quoted 8 lines

Quoted 8 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's ignore

Quoted 1 line

the DMA TX completion transaction for now - and call it a possible

future

Quoted 1 line

optimization for specific use cases. So it seems that some form of

completion call

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

is unavoidable.

Agree to leave this aside, there is in theory a potential optimization, but
unlikely to be of large value.

One more thing: When using DMA to pass on packets into a guest, there could

be a

Quoted 1 line

delay from the DMA completes until the guest is signaled. Is there any CPU

cache

Quoted 1 line

hotness regarding the guest's access to the packet data to consider here? I.e. if

we

Quoted 1 line

wait signaling the guest, the packet data may get cold.

Interesting question; we can likely spawn a new thread around this topic! In short, it depends on how/where the DMA hardware writes the copy.

With technologies like DDIO, the "dest" part of the copy will be in LLC. The core reading the
dest data will benefit from the LLC locality (instead of snooping it from a remote core's L1/L2).

Delays in notifying the guest could result in LLC capacity eviction, yes. The application layer decides how often/promptly to check for completions, and notify the guest of them. Calling the function more often will result in less delay in that portion of the pipeline.

Overall, there are caching benefits with DMA acceleration, and the application can control
the latency introduced between dma-completion done in HW, and Guest vring update.

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, April 7, 2022 3:40 PM
To: Maxime Coquelin <[email protected]>; Van Haaren, Harry <[email protected]>; Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; [email protected]; [email protected]; Mcnamara, John <[email protected]>; O'Driscoll, Tim <[email protected]>; Finn, Emma <[email protected]>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion

Quoted 9 lines · Maxime Coquelin

Hi Harry,

Quoted 6 lines · Harry

Hi OVS & DPDK, Maintainers & Community,

Top posting overview of discussion as replies to thread become slower:
perhaps it is a good time to review and plan for next steps?

 From my perspective, it those most vocal in the thread seem to be in favour

of the clean

Quoted 1 line

Quoted 1 line

rx/tx split ("defer work"), with the tradeoff that the application must be

aware of handling

Quoted 1 line

Quoted 1 line

the async DMA completions. If there are any concerns opposing upstreaming

of this method,

Quoted 1 line

Quoted 1 line

please indicate this promptly, and we can continue technical discussions here

now.

Quoted 3 lines

Wasn't there some discussions about handling the Virtio completions with the DMA engine? With that, we wouldn't need the deferral of work.

+1

With the virtio completions handled by DMA itself, the vhost port turns almost into a real HW NIC. With that we will not need any extra manipulations from the OVS side, i.e. no need to defer any work while maintaining clear split between rx and tx operations.

I'd vote for that.

Quoted 3 lines

Thanks,
Maxime

Quoted 1 line

Quoted 1 line

In absence of continued technical discussion here, I suggest Sunil and Ian

collaborate on getting

Quoted 1 line

Quoted 1 line

the OVS Defer-work approach, and DPDK VHost Async patchsets available on

GitHub for easier

Quoted 1 line

Quoted 1 line

consumption and future development (as suggested in slides presented on

last call).

Quoted 5 lines

Quoted 5 lines

Regards, -Harry

No inline-replies below; message just for context.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

-----Original Message-----
From: Van Haaren, Harry
Sent: Wednesday, March 30, 2022 10:02 AM
To: Morten Brørup <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya

Maximets

Quoted 1 line

Quoted 1 line

Quoted 1 line

<[email protected]>; [email protected]; [email protected];

Mcnamara,

Quoted 1 line

Quoted 1 line

Quoted 1 line

John <[email protected]>; O'Driscoll, Tim

<[email protected]>;

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Finn, Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 8 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 8:59 PM
To: Van Haaren, Harry <[email protected]>; Richardson, Bruce <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Pai G, Sunil <[email protected]>; Stokes, Ian <[email protected]>; Hu, Jiayu <[email protected]>; Ferriter, Cian <[email protected]>; Ilya

Maximets

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

<[email protected]>; [email protected]; [email protected];

Mcnamara,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

John

Quoted 1 line

<[email protected]>; O'Driscoll, Tim <[email protected]>;

Finn,

Quoted 161 lines

Quoted 161 lines

Quoted 161 lines

Quoted 161 lines

Emma <[email protected]>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion

Quoted 157 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.46

Quoted 24 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 6:14 PM

Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 19.03

Quoted 5 lines · Morten Brørup

Quoted 5 lines

From: Maxime Coquelin [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 18.24

Hi Morten,

Quoted 8 lines

Quoted 8 lines · Morten Brørup

Quoted 7 lines

Quoted 7 lines

From: Van Haaren, Harry [mailto:[email protected]]
Sent: Tuesday, 29 March 2022 15.02

Quoted 4 lines

From: Morten Brørup <[email protected]>
Sent: Tuesday, March 29, 2022 1:51 PM

Having thought more about it, I think that a completely

different

Quoted 3 lines

Quoted 3 lines

architectural approach is required:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Many of the DPDK Ethernet PMDs implement a variety of RX

and TX

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

packet burst functions, each optimized for different CPU vector instruction sets. The availability of a DMA engine should be

treated

Quoted 1 line

Quoted 1 line

the same way. So I suggest that PMDs copying packet contents,

e.g.

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

Quoted 1 line

memif, pcap, vmxnet3, should implement DMA optimized RX and TX

packet

Quoted 5 lines

Quoted 5 lines

burst functions.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Similarly for the DPDK vhost library.

In such an architecture, it would be the application's job

to

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

allocate DMA channels and assign them to the specific PMDs that

should

Quoted 1 line

Quoted 1 line

use them. But the actual use of the DMA channels would move

down

Quoted 7 lines

Quoted 7 lines

below

Quoted 6 lines

Quoted 6 lines

the application and into the DPDK PMDs and libraries.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Med venlig hilsen / Kind regards,
-Morten Brørup

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Hi Morten,

That's *exactly* how this architecture is designed &

implemented.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

1.    The DMA configuration and initialization is up to the

application

Quoted 2 lines

Quoted 2 lines

(OVS).

Quoted 1 line

Quoted 1 line

2.    The VHost library is passed the DMA-dev ID, and its

new

Quoted 4 lines

Quoted 4 lines

async

Quoted 3 lines

Quoted 3 lines

rx/tx APIs, and uses the DMA device to accelerate the copy.

Quoted 2 lines

Quoted 2 lines

Looking forward to talking on the call that just started.

Quoted 8 lines

Quoted 8 lines

Regards, -

Quoted 4 lines

Quoted 4 lines

Harry

Quoted 3 lines

Quoted 1 line

OK, thanks - as I said on the call, I haven't looked at the

patches.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Then, I suppose that the TX completions can be handled in the

TX

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

function, and the RX completions can be handled in the RX

function,

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

just like the Ethdev PMDs handle packet descriptors:

Quoted 3 lines

TX_Burst(tx_packet_array):
1.    Clean up descriptors processed by the NIC chip. -->

Process

Quoted 3 lines

Quoted 3 lines

TX

Quoted 2 lines

Quoted 2 lines

DMA channel completions. (Effectively, the 2nd pipeline stage.)

Quoted 1 line

2.    Pass on the tx_packet_array to the NIC chip

descriptors. --

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Pass

Quoted 1 line

on the tx_packet_array to the TX DMA channel. (Effectively, the

1st

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

pipeline stage.)

The problem is Tx function might not be called again, so

enqueued

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

packets in 2. may never be completed from a Virtio point of

view.

Quoted 2 lines

Quoted 2 lines

IOW,

Quoted 1 line

Quoted 1 line

the packets will be copied to the Virtio descriptors buffers,

but

Quoted 4 lines

Quoted 4 lines

the

Quoted 3 lines

Quoted 1 line

descriptors will not be made available to the Virtio driver.

In that case, the application needs to call TX_Burst()

periodically

Quoted 1 line

Quoted 1 line

with an empty array, for completion purposes.

This is what the "defer work" does at the OVS thread-level, but instead of
"brute-forcing" and *always* making the call, the defer work concept tracks
*when* there is outstanding work (DMA copies) to be completed ("deferred work")
and calls the generic completion function at that point.

So "defer work" is generic infrastructure at the OVS thread level to handle
work that needs to be done "later", e.g. DMA completion handling.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

Or some sort of TX_Keepalive() function can be added to the DPDK

library, to handle DMA completion. It might even handle multiple

DMA

Quoted 2 lines

Quoted 2 lines

channels, if convenient - and if possible without locking or other weird complexity.

That's exactly how it is done, the VHost library has a new API added, which allows
for handling completions. And in the "Netdev layer" (~OVS ethdev abstraction)
we add a function to allow the OVS thread to do those completions in a new
Netdev-abstraction API called "async_process" where the completions can be checked.

The only method to abstract them is to "hide" them somewhere that will always be
polled, e.g. an ethdev port's RX function.  Both V3 and V4 approaches use this method.
This allows "completions" to be transparent to the app, at the tradeoff to having bad
separation  of concerns as Rx and Tx are now tied-together.

The point is, the Application layer must *somehow * handle of completions.
So fundamentally there are 2 options for the Application level:

A) Make the application periodically call a "handle completions" function
    A1) Defer work, call when needed, and track "needed" at app layer, and calling into vhost txq complete as required. Elegant in that "no work" means "no cycles spent" on checking DMA completions.
    A2) Brute-force-always-call, and pay some overhead when not required.
            Cycle-cost in "no work" scenarios. Depending on # of vhost queues, this adds up as polling required *per vhost txq*. Also note that "checking DMA completions" means taking a virtq-lock, so this "brute-force" can needlessly increase x-thread contention!

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

A side note: I don't see why locking is required to test for DMA

completions.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 2 lines

rte_dma_vchan_status() is lockless, e.g.:

https://elixir.bootlin.com/dpdk/latest/source/drivers/dma/ioat/ioat_dmadev.c#L

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

56

Quoted 1 line

0

Correct, DMA-dev is "ethdev like"; each DMA-id can be used in a lockfree

manner

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

from a single thread.

The locks I refer to are at the OVS-netdev level, as virtq's are shared across

OVS's

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

dataplane threads.
So the "M to N" comes from M dataplane threads to N virtqs, hence

requiring

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

some locking.

Quoted 11 lines

Quoted 10 lines

B) Hide completions and live with the complexity/architectural sacrifice of mixed-RxTx.
    Various downsides here in my opinion, see the slide deck presented earlier today for a summary.

In my opinion, A1 is the most elegant solution, as it has a clean separation of concerns, does not  cause
avoidable contention on virtq locks, and spends no cycles when there is no completion work to do.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

Thank you for elaborating, Harry.

Thanks for part-taking in the discussion & providing your insight!

Quoted 1 line

I strongly oppose against hiding any part of TX processing in an RX function.

It

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

is just

Quoted 3 lines

wrong in so many ways!

I agree that A1 is the most elegant solution. And being the most elegant

solution, it

Quoted 1 line

is probably also the most future proof solution. :-)

I think so too, yes.

Quoted 1 line

I would also like to stress that DMA completion handling belongs in the

DPDK

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

library, not in the application. And yes, the application will be required to

call

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

some

Quoted 1 line

"handle DMA completions" function in the DPDK library. But since the

application

Quoted 1 line

already knows that it uses DMA, the application should also know that it

needs

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

to

Quoted 1 line

call this extra function - so I consider this requirement perfectly acceptable.

Agree here.

Quoted 1 line

I prefer if the DPDK vhost library can hide its inner workings from the

application,

Quoted 1 line

and just expose the additional "handle completions" function. This also

means

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

that

Quoted 2 lines

the inner workings can be implemented as "defer work", or by some other algorithm. And it can be tweaked and optimized later.

Yes, the choice in how to call the handle_completions function is Application layer.
For OVS we designed Defer Work, V3 and V4. But it is an App level choice,

and

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

every
application is free to choose its own method.

Quoted 1 line

Thinking about the long term perspective, this design pattern is common

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

both

Quoted 2 lines

the vhost library and other DPDK libraries that could benefit from DMA (e.g. vmxnet3 and pcap PMDs), so it could be abstracted into the DMA library or

a

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

separate library. But for now, we should focus on the vhost use case, and

just

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

keep

Quoted 1 line

the long term roadmap for using DMA in mind.

Totally agree to keep long term roadmap in mind; but I'm not sure we can refactor
logic out of vhost. When DMA-completions arrive, the virtQ needs to be updated;
this causes a tight coupling between the DMA completion count, and the

vhost

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

library.

As Ilya raised on the call yesterday, there is an "in_order" requirement in the vhost
library, that per virtq the packets are presented to the guest "in order" of enqueue.
(To be clear, *not* order of DMA-completion! As Jiayu mentioned, the Vhost library
handles this today by re-ordering the DMA completions.)

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Rephrasing what I said on the conference call: This vhost design will

become

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

the

Quoted 1 line

common design pattern for using DMA in DPDK libraries. If we get it wrong,

we

Quoted 33 lines

Quoted 33 lines

Quoted 33 lines

are

Quoted 1 line

stuck with it.

Agree, and if we get it right, then we're stuck with it too! :)

Quoted 27 lines

Quoted 27 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Here is another idea, inspired by a presentation at one of the

DPDK

Quoted 8 lines

Quoted 8 lines

Userspace conferences. It may be wishful thinking, though:

Quoted 2 lines

Add an additional transaction to each DMA burst; a special

transaction containing the memory write operation that makes the descriptors available to the Virtio driver.

Quoted 1 line

That is something that can work, so long as the receiver is

operating

Quoted 2 lines

Quoted 2 lines

in
polling mode. For cases where virtio interrupts are enabled, you

still

Quoted 8 lines

Quoted 6 lines

need
to do a write to the eventfd in the kernel in vhost to signal the virtio
side. That's not something that can be offloaded to a DMA engine, sadly, so
we still need some form of completion call.

I guess that virtio interrupts is the most widely deployed scenario,

so let's ignore

Quoted 1 line

the DMA TX completion transaction for now - and call it a possible

future

Quoted 1 line

optimization for specific use cases. So it seems that some form of

completion call

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

is unavoidable.

Agree to leave this aside, there is in theory a potential optimization, but
unlikely to be of large value.

One more thing: When using DMA to pass on packets into a guest, there

could

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

be a

Quoted 1 line

delay from the DMA completes until the guest is signaled. Is there any CPU

cache

Quoted 1 line

hotness regarding the guest's access to the packet data to consider here?

I.e. if

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

we

Quoted 1 line

wait signaling the guest, the packet data may get cold.

Interesting question; we can likely spawn a new thread around this topic! In short, it depends on how/where the DMA hardware writes the copy.

With technologies like DDIO, the "dest" part of the copy will be in LLC. The

core

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

reading the
dest data will benefit from the LLC locality (instead of snooping it from a

remote

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

core's L1/L2).

Delays in notifying the guest could result in LLC capacity eviction, yes. The application layer decides how often/promptly to check for completions, and notify the guest of them. Calling the function more often will result in

less

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

delay in that portion of the pipeline.

Overall, there are caching benefits with DMA acceleration, and the

application

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

can control
the latency introduced between dma-completion done in HW, and Guest

vring

Quoted 3 lines

Quoted 2 lines

Quoted 1 line

update.