NAPI as kobject proposal

The NAPI interface structure in current kernels is managed by the driver. As part of receive packet steering there is a requirement to add an additional parameter to this for the CPU map. And this map needs to have an API to set it.

The right way to do this in the kernel model is to make NAPI into a kobject and associate it back with the network device (parent). This isn't wildly difficult but does change some of the API for network device drivers because:
1. They need to handle another possible error on setup
2. NAPI object needs to be dynamically allocated separately (not as part of netdev_priv)
3. Driver should pass index that can be uses as part of name (easier than scanning)

Eventually, there will be:
/sys/class/net/eth0/napi0/
weight
cpumap

So here is a starting point patch that shows how the API might look like.

---
 include/linux/netdevice.h |   20 ++++++++++++++------
 net/core/dev.c            |   28 ++++++++++++++++++++++++++--
 2 files changed, 40 insertions(+), 8 deletions(-)

include/linux/netdevice.h +5 -2

@@ -378,6 +378,8 @@ struct napi_struct {
 	struct list_head	dev_list;
 	struct sk_buff		*gro_list;
 	struct sk_buff		*skb;
+
+	struct kobject		kobj;
 };
 
 enum {

@@ -1037,25 +1039,31 @@ static inline void *netdev_priv(const st
 #define SET_NETDEV_DEVTYPE(net, devtype)	((net)->dev.type = (devtype))
 
 /**
- *	netif_napi_add - initialize a napi context
+ *	netif_napi_init - initialize a napi context
  *	@dev:  network device
  *	@napi: napi context
+ *	@index: queue number
  *	@poll: polling function
  *	@weight: default weight
  *
- * netif_napi_add() must be used to initialize a napi context prior to calling
+ * netif_napi_init() must be used to create a napi context prior to calling

Quoted 5 lines · Stephen Hemminger

The NAPI interface structure in current kernels is managed by the driver. As part of receive packet steering there is a requirement to add an additional parameter to this for the CPU map. And this map needs to have an API to set it.

Quoted 9 lines

The right way to do this in the kernel model is to make NAPI into a kobject and associate it back with the network device (parent). This isn't wildly difficult but does change some of the API for network device drivers because:
1. They need to handle another possible error on setup
2. NAPI object needs to be dynamically allocated separately (not as part of netdev_priv)
3. Driver should pass index that can be uses as part of name (easier than scanning)

Quoted 6 lines · Ben Hutchings

Quoted 5 lines · Stephen Hemminger

The NAPI interface structure in current kernels is managed by the driver. As part of receive packet steering there is a requirement to add an additional parameter to this for the CPU map. And this map needs to have an API to set it.

Quoted 9 lines

Quoted 9 lines

The right way to do this in the kernel model is to make NAPI into a kobject and associate it back with the network device (parent). This isn't wildly difficult but does change some of the API for network device drivers because:
1. They need to handle another possible error on setup
2. NAPI object needs to be dynamically allocated separately (not as part of netdev_priv)
3. Driver should pass index that can be uses as part of name (easier than scanning)

4. Lifetime rules become oh-so-fscking-interesting?

Quoted 15 lines · Stephen Hemminger

The NAPI interface structure in current kernels is managed by the driver. As part of receive packet steering there is a requirement to add an additional parameter to this for the CPU map. And this map needs to have an API to set it.

The right way to do this in the kernel model is to make NAPI into a kobject and associate it back with the network device (parent). This isn't wildly difficult but does change some of the API for network device drivers because:
1. They need to handle another possible error on setup
2. NAPI object needs to be dynamically allocated separately (not as part of netdev_priv)
3. Driver should pass index that can be uses as part of name (easier than scanning)

Quoted 4 lines

Eventually, there will be:
/sys/class/net/eth0/napi0/
weight
cpumap

I think the NAPI objects should be created as children of a bus device and then linked from the net device directories, e.g.

/sys/devices/.../ net/eth0/

                           napi0 -> ../../napi/napi0
                      eth1/
                           napi0 -> ../../napi/napi0
                  napi/napi0/
                             weight
                             cpumap

As part of receive packet steering there is a requirement to add an additional parameter to this for the CPU map.

From: Stephen Hemminger <[email protected]>
Date: Fri, 29 Jan 2010 10:18:39 -0800

Quoted 2 lines

As part of receive packet steering there is a requirement to add an additional parameter to this for the CPU map.

Hmmm, where did this come from?

The RPS maps are per-device.

I think I vaguely recall you "suggesting" that the RPS maps become per-NAPI.

But, firstly, I didn't see any movement in that part of the discussion.

And, secondly, I don't think this makes any sense at all.

Things are already overly complicated as it is. Having the user know what traffic goes to a particular RX queue (ie. NAPI instance) and set the RPS map in some way specific to that RX queue is over the top.

If the issue is the case of sharing a NAPI instance between two devices, there are a few other ways to deal with this.

One I would suggest is to simply clone the RPS map amongst the devices sharing a NAPI instance.

I currently see NAPI kobjects is just an over-abstraction for a perceived need rather than a real one.

The usage model I see is wanting to have:
1. only some cores being used for receive traffic on single Rx devices (NAPI)
2. only some cores being used for receive traffic on legacy devices (non-NAPI)
3. being able to configure a set of cpus with same IRQ/cache when doing Rx multi-queue. Assign MSI-X IRQ per core and allow both HT on core to split that RX traffic.

All this should be manageable by some user utility like irqbalance.

#1 and #2 argue for a per device map (like irq_affinity) but #3 is harder; not sure the right API for that.