Skip to contentSkip to conversation

DPDK and HW offloads

DISCUSSION Stephen Hemminger 8 participants 12 messages Mar 18, 2016 – Mar 23, 2016

lore
Share & tools
lore
#0 lore raw

As I look at how the ethernet device interface in DPDK has exploded in complexity; it makes life very hard for end users. The goal has been to enable all the cool hardware features, but it has put blinders on the driver devlopers; they are ignoring the fact that real applications can't just work on one kind of hardware.

The DPDK is doing a terrible job at providing abstractions. There needs to be a real generic set of operations, and every hardware offload feature must:
* have a clear well defined API
* if feature is not available in software, then the DPDK must provide a software equivalent feature.
* any difference in API must be hidden from application.
* no compile config options about offload.
* tests and documentation must work for both hw and sw version

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the filter/offload/ptype stuff from DPDK!

#1 lore raw

2016-03-18 10:16, Stephen Hemminger:

Quoted 1 line

As I look at how the ethernet device interface in DPDK has exploded in complexity;

Yes I would like to start addressing this complexity in 16.07.

Quoted 3 lines

it makes life very hard for end users. The goal has been to enable all the cool hardware features, but it has put blinders on the driver devlopers; they are ignoring the fact that real applications can't just work on one kind of hardware.

+1

Quoted 3 lines

The DPDK is doing a terrible job at providing abstractions. There needs to be a real generic set of operations, and every hardware offload feature must:
* have a clear well defined API

+1

Quoted 2 lines

* if feature is not available in software, then the DPDK must provide a software equivalent feature.

I'm not against this idea. Looking for more opinions.

Quoted 7 lines

* any difference in API must be hidden from application.
* no compile config options about offload.
* tests and documentation must work for both hw and sw version

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

#2 lore raw
Quoted 13 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Saturday, March 19, 2016 2:01 AM
To: Stephen Hemminger <[email protected]>
Cc: Zhang, Helin <[email protected]>; [email protected]
Subject: Re: DPDK and HW offloads

2016-03-18 10:16, Stephen Hemminger:

Quoted 2 lines

As I look at how the ethernet device interface in DPDK has exploded in complexity;

Quoted 8 lines

Yes I would like to start addressing this complexity in 16.07.

Quoted 3 lines

it makes life very hard for end users. The goal has been to enable all the cool hardware features, but it has put blinders on the driver devlopers; they are ignoring the fact that real applications can't just work on

one kind of hardware.

+1

Agree that DPDK needs to add common APIs from application's point of view.

Quoted 20 lines
Quoted 3 lines

The DPDK is doing a terrible job at providing abstractions. There needs to be a real generic set of operations, and every hardware offload

feature must:

Quoted 1 line

* have a clear well defined API

+1

Quoted 2 lines

* if feature is not available in software, then the DPDK must provide a software equivalent feature.

I'm not against this idea. Looking for more opinions.

Quoted 7 lines

* any difference in API must be hidden from application.
* no compile config options about offload.
* tests and documentation must work for both hw and sw version

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

Quoted 8 lines

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

#3 lore raw

2016-03-20 14:17, Zhang, Helin:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

2016-03-18 10:16, Stephen Hemminger:

Quoted 3 lines

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

Quoted 8 lines
Quoted 3 lines

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

Yes we cannot access to every features of a device through generic API. Until now we were trying to add an ethdev API for every features even if it is used by only one driver.
I think we should allow a direct access to the driver by the applications and work on generic API only for common features.

#4 lore raw
Quoted 20 lines · Thomas Monjalon

2016-03-20 14:17, Zhang, Helin:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

2016-03-18 10:16, Stephen Hemminger:

Quoted 3 lines

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

Quoted 8 lines
Quoted 3 lines

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

Quoted 5 lines

Yes we cannot access to every features of a device through generic API. Until now we were trying to add an ethdev API for every features even if it is used by only one driver.
I think we should allow a direct access to the driver by the applications and work on generic API only for common features.

Definite +1.
I think that we need to start pushing driver-specific functionality to get exposed via a driver's header files. That allow users who want to extract the max functionality from a particular NIC to do so via those APIs calls, while not polluting the generic ethdev layer.

On the other hand, I don't like the idea of dpdk_netdev. I think we can work within the existing rte_eth_dev framework.

/Bruce

#5 lore raw
Quoted 27 lines · Bruce Richardson
Quoted 20 lines · Thomas Monjalon

2016-03-20 14:17, Zhang, Helin:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

2016-03-18 10:16, Stephen Hemminger:

Quoted 3 lines

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

Quoted 8 lines
Quoted 3 lines

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

Quoted 5 lines

Yes we cannot access to every features of a device through generic API. Until now we were trying to add an ethdev API for every features even if it is used by only one driver.
I think we should allow a direct access to the driver by the applications and work on generic API only for common features.

Quoted 6 lines

Definite +1.
I think that we need to start pushing driver-specific functionality to get exposed via a driver's header files. That allow users who want to extract the max functionality from a particular NIC to do so via those APIs calls, while not polluting the generic ethdev layer.

What sort of requirements on ABI/API compatibility would this place on the drivers? I would hope that it would be treated like any other public API within DPDK. I don't think this would be too onerous, but it would require that the drivers be designed to deal with it. (I.e. don't just expose any old internal driver function).

Quoted 5 lines

On the other hand, I don't like the idea of dpdk_netdev. I think we can work within the existing rte_eth_dev framework.

/Bruce

#6 lore raw
Quoted 32 lines · Kyle Larose
Quoted 24 lines · Bruce Richardson
Quoted 18 lines · Thomas Monjalon

2016-03-20 14:17, Zhang, Helin:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

2016-03-18 10:16, Stephen Hemminger:

Quoted 3 lines

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

Quoted 7 lines
Quoted 3 lines

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

Quoted 5 lines

Yes we cannot access to every features of a device through generic API. Until now we were trying to add an ethdev API for every features even if it is used by only one driver.
I think we should allow a direct access to the driver by the applications and work on generic API only for common features.

Quoted 6 lines

Definite +1.
I think that we need to start pushing driver-specific functionality to get exposed via a driver's header files. That allow users who want to extract the max functionality from a particular NIC to do so via those APIs calls, while not polluting the generic ethdev layer.

Quoted 5 lines

What sort of requirements on ABI/API compatibility would this place on the drivers? I would hope that it would be treated like any other public API within DPDK. I don't think this would be too onerous, but it would require that the drivers be designed to deal with it. (I.e. don't just expose any old internal driver function).

Why not to implement one simple API with variable arguments, just like syscall ioctl() does. And drivers implement it's specific hardware features with a feature bit param, and other needed variable arguments.

Thanks,
Michael

Quoted 5 lines
Quoted 5 lines

On the other hand, I don't like the idea of dpdk_netdev. I think we can work within the existing rte_eth_dev framework.

/Bruce

#7 lore raw
Quoted 39 lines · Michael
Quoted 32 lines · Kyle Larose
Quoted 24 lines · Bruce Richardson
Quoted 18 lines · Thomas Monjalon

2016-03-20 14:17, Zhang, Helin:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

2016-03-18 10:16, Stephen Hemminger:

Quoted 3 lines

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

Quoted 7 lines
Quoted 3 lines

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

Quoted 5 lines

Yes we cannot access to every features of a device through generic API. Until now we were trying to add an ethdev API for every features even if it is used by only one driver.
I think we should allow a direct access to the driver by the applications and work on generic API only for common features.

Quoted 6 lines

Definite +1.
I think that we need to start pushing driver-specific functionality to get exposed via a driver's header files. That allow users who want to extract the max functionality from a particular NIC to do so via those APIs calls, while not polluting the generic ethdev layer.

Quoted 5 lines

What sort of requirements on ABI/API compatibility would this place on the drivers? I would hope that it would be treated like any other public API within DPDK. I don't think this would be too onerous, but it would require that the drivers be designed to deal with it. (I.e. don't just expose any old internal driver function).

Quoted 6 lines

Why not to implement one simple API with variable arguments, just like syscall ioctl() does. And drivers implement it's specific hardware features with a feature bit param, and other needed variable arguments.

Thanks,
Michael

A very much dislike that idea.
* It makes the code much harder to read as you have to closely examine all the parameters to work out what a function call is actually meant to do.
* It makes it much harder to see that you have an implicit dependency on a specific device. Having to include a driver specific header file e.g. i40e.h, and call a function named e.g. i40e_do_magic_stuff(), makes it pretty explicit that you have a dependency on i40e-based hardware
* It prevents the compiler from doing type-checking on parameters and informing you of little inconsistencies.

For all these reasons, I prefer the device-specific functions option. However, at the same time, we also need to ensure we have a reasonable set of generic APIs so that the cases where users are forced to drop down to the lower-level device-specific primitives are reduced.

Regards,
/Bruce

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

On the other hand, I don't like the idea of dpdk_netdev. I think we can work within the existing rte_eth_dev framework.

/Bruce

#8 lore raw
Quoted 23 lines · Bruce Richardson
Quoted 7 lines · Michael

Why not to implement one simple API with variable arguments, just like syscall ioctl() does. And drivers implement it's specific hardware features with a feature bit param, and other needed variable arguments.

Thanks,
Michael

A very much dislike that idea.
* It makes the code much harder to read as you have to closely examine all the
parameters to work out what a function call is actually meant to do.
* It makes it much harder to see that you have an implicit dependency on a specific device. Having to include a driver specific header file e.g.
i40e.h,
and call a function named e.g. i40e_do_magic_stuff(), makes it pretty
explicit
that you have a dependency on i40e-based hardware
* It prevents the compiler from doing type-checking on parameters and informing
you of little inconsistencies.

Quoted 8 lines

For all these reasons, I prefer the device-specific functions option. However,
at the same time, we also need to ensure we have a reasonable set of generic
APIs so that the cases where users are forced to drop down to the lower-level
device-specific primitives are reduced.

+1

Jay

#9 lore raw
Quoted 35 lines · On Tue, 22 Mar 2016 07:19:01 -0500 Jay Rolette <[email protected]> wrote:
Quoted 23 lines · Bruce Richardson
Quoted 7 lines · Michael

Why not to implement one simple API with variable arguments, just like syscall ioctl() does. And drivers implement it's specific hardware features with a feature bit param, and other needed variable arguments.

Thanks,
Michael

A very much dislike that idea.
* It makes the code much harder to read as you have to closely examine all the
parameters to work out what a function call is actually meant to do.
* It makes it much harder to see that you have an implicit dependency on a specific device. Having to include a driver specific header file e.g.
i40e.h,
and call a function named e.g. i40e_do_magic_stuff(), makes it pretty
explicit
that you have a dependency on i40e-based hardware
* It prevents the compiler from doing type-checking on parameters and informing
you of little inconsistencies.

Quoted 8 lines

For all these reasons, I prefer the device-specific functions option. However,
at the same time, we also need to ensure we have a reasonable set of generic
APIs so that the cases where users are forced to drop down to the lower-level
device-specific primitives are reduced.

Quoted 1 line

+1

I prefer the OO model where there is a generic network interface that provides a uniform set of features, and if a specific hw device provides a more efficient implementation of a feature, that is hidden (inheritance?) such that there is no difference API.

The DPDK has been hardware vendor driven up until this point.
But from a software point of view, it is more important to provide good API than use DPDK as a hardware POC environment.

#10 lore raw
Quoted 40 lines · Stephen Hemminger
Quoted 35 lines · On Tue, 22 Mar 2016 07:19:01 -0500 Jay Rolette <[email protected]> wrote:
Quoted 23 lines · Bruce Richardson
Quoted 7 lines · Michael

Why not to implement one simple API with variable arguments, just like syscall ioctl() does. And drivers implement it's specific hardware features with a feature bit param, and other needed variable arguments.

Thanks,
Michael

A very much dislike that idea.
* It makes the code much harder to read as you have to closely examine all the
parameters to work out what a function call is actually meant to do.
* It makes it much harder to see that you have an implicit dependency on a specific device. Having to include a driver specific header file e.g.
i40e.h,
and call a function named e.g. i40e_do_magic_stuff(), makes it pretty
explicit
that you have a dependency on i40e-based hardware
* It prevents the compiler from doing type-checking on parameters and informing
you of little inconsistencies.

Quoted 8 lines

For all these reasons, I prefer the device-specific functions option. However,
at the same time, we also need to ensure we have a reasonable set of generic
APIs so that the cases where users are forced to drop down to the lower-level
device-specific primitives are reduced.

Quoted 1 line

+1

Quoted 8 lines

I prefer the OO model where there is a generic network interface that provides a uniform set of features, and if a specific hw device provides a more efficient implementation of a feature, that is hidden (inheritance?) such that there is no difference API.

The DPDK has been hardware vendor driven up until this point.
But from a software point of view, it is more important to provide good API than use DPDK as a hardware POC environment.

+1

#11 lore raw
In reply to Bruce Richardson
Quoted 43 lines · Bruce
Quoted 42 lines · Michael
Quoted 32 lines · Kyle Larose
Quoted 24 lines · Bruce Richardson
Quoted 18 lines · Thomas Monjalon

2016-03-20 14:17, Zhang, Helin:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

2016-03-18 10:16, Stephen Hemminger:

Quoted 3 lines

Right now, all those offload features are pretty much unusable in a real product without lots and lots of extra codes and huge bug surface. It bothers me enough that I would recommend removing much of the

filter/offload/ptype stuff from DPDK!

One of the biggest challenge is to think about a good filtering API. The offloading has some interaction with the mbuf struct.

Quoted 7 lines
Quoted 3 lines

I would like to suggest rewriting ethdev API by keeping it as is for some time for compatibility while creating a new one. What about the prefix dpdk_netdev_ to progressively replace rte_eth_dev?

I totally agree with to add new and generic APIs for user applications. But I don't think we need to remove all current APIs. Generic APIs may not support all advanced hardware features, while specific APIs can. Why not support all? One generic APIs for common users, and others APIs for advanced users.

Quoted 5 lines

Yes we cannot access to every features of a device through generic API. Until now we were trying to add an ethdev API for every features even if it is used by only one driver.
I think we should allow a direct access to the driver by the applications and work on generic API only for common features.

Quoted 6 lines

Definite +1.
I think that we need to start pushing driver-specific functionality to get exposed via a driver's header files. That allow users who want to extract the max functionality from a particular NIC to do so via those APIs calls, while not polluting the generic ethdev layer.

Quoted 5 lines

What sort of requirements on ABI/API compatibility would this place on the drivers? I would hope that it would be treated like any other public API within DPDK. I don't think this would be too onerous, but it would require that the drivers be designed to deal with it. (I.e. don't just expose any old internal driver function).

Why not to implement one simple API with variable arguments, just like syscall ioctl() does. And drivers implement it's specific hardware features with a feature bit param, and other needed variable arguments.

Quoted 5 lines
Quoted 2 lines

Thanks,
Michael

A very much dislike that idea.
* It makes the code much harder to read as you have to closely examine all the parameters to work out what a function call is actually meant to do.

It's not a big deal, if we have a document.

Quoted 4 lines

* It makes it much harder to see that you have an implicit dependency on a specific device. Having to include a driver specific header file e.g. i40e.h, and call a function named e.g. i40e_do_magic_stuff(), makes it pretty explicit that you have a dependency on i40e-based hardware

Software does not want to bind to specific hardware I think, what about the transportability?

Quoted 2 lines

* It prevents the compiler from doing type-checking on parameters and informing you of little inconsistencies.

Maybe, we could do self-check for the parameters I think.

Quoted 5 lines

For all these reasons, I prefer the device-specific functions option. However, at the same time, we also need to ensure we have a reasonable set of generic APIs so that the cases where users are forced to drop down to the lower-level device-specific primitives are reduced.

For software, it do not care which hardware it is, it only cares about what ability you have.

Thanks,
Michael

Quoted 9 lines

Regards,
/Bruce

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

On the other hand, I don't like the idea of dpdk_netdev. I think we can work within the existing rte_eth_dev framework.

/Bruce