netkit: Support for io_uring zero-copy and AF_XDP
Subsystems (11)
Daniel Borkmann
maintainerv8 cover netkit: Support for io_uring zero-copy and AF_XDP
Containers use virtual netdevs to route traffic from a physical netdev in the host namespace. They do not have access to the physical netdev in the host and thus can't use memory providers or AF_XDP that require reconfiguring/restarting queues in the physical netdev.
This patchset adds the concept of queue leasing to virtual netdevs that allow containers to use memory providers and AF_XDP at native speed. Leased queues are bound to a real queue in a physical netdev and act as a proxy.
Memory providers and AF_XDP operations take an ifindex and queue id, so containers would pass in an ifindex for a virtual netdev and a queue id of a leased queue, which then gets proxied to the underlying real queue.
We have implemented support for this concept in netkit and tested the latter against Nvidia ConnectX-6 (mlx5) as well as Broadcom BCM957504 (bnxt_en) 100G NICs. For more details see the individual patches.
v7->v8:
- Rework and refactor net_mp_{open,close}_rxq patch (Jakub)
- Moved queue lease netlink command from env into test (Jakub)
- Support netns_id also in queue create call and not only dump
- Rebase and retested everything with mlx5 + bnxt_env6->v7:
- Add xsk_dev_queue_valid real_num_rx_queues check given bound xs->queue_id could be from a TX queue (AI review bot)
- Fix up exception path in queue leasing selftest (AI review bot)
- Rebase and retested everything with mlx5 + bnxt_env5->v6:
- Fix nest_queue test in netdev_nl_queue_fill_one (Jakub/AI review bot)
- Fix netdev notifier locking leak (Jakub/AI review bot)
- Drop NETREG_UNREGISTERING WARN_ON_ONCE to avoid confusion (Stan)
- Remove slipped-in .gitignore cruft in net selftest (Stan)
- Fix Pylint warnings in net selftest (Jakub)
- Rebase and retested everything with mlx5 + bnxt_env4->v5:
- Rework of the core API into queue-create op (Jakub)
- Rename from queue peering to queue leasing (Jakub)
- Add net selftests for queue leasing (Stan, Jakub)
- Move netkit_queue_get_dma_dev into core (Jakub)
- Dropped netkit_get_channels (Jakub)
- Moved ndo_queue_create back to return index or error (Jakub)
- Inline __netdev_rx_queue_{peer,unpeer} helpers (Jakub)
- Adding helpers in patches where they are used (Jakub)
- Undo inline for netdev_put_lock (Jakub)
- Factoring out checks whether device can lease (Jakub)
- Fix up return codes in netdev_nl_bind_queue_doit (Jakub)
- Reject when AF_XDP or mp already bound (Jakub)
- Switch some error cases to NL_SET_BAD_ATTR() (Jakub)
- Rebase and retested everything with mlx5 + bnxt_env3->v4:
- ndo_queue_create store dst queue via arg (Nikolay)
- Small nits like a spelling issue + rev xmas (Nikolay)
- admin-perm flag in bind-queue spec (Jakub)
- Fix potential ABBA deadlock situation in bind (Jakub, Paolo, Stan)
- Add a peer dev_tracker to not reuse the sysfs one (Jakub)
- New patch (12/14) to handle the underlying device going away (Jakub)
- Improve commit message on queue-get (Jakub)
- Do not expose phys dev info from container on queue-get (Jakub)
- Add netif_put_rx_queue_peer_locked to simplify code (Stan)
- Rework xsk handling to simplify the code and drop a few patches
- Rebase and retested everything with mlx5 + bnxt_env2->v3:
- Use netdev_ops_assert_locked instead of netdev_assert_locked (syzbot)
- Add missing netdev_lockdep_set_classes in netkitv1->v2:
- Removed bind sample ynl code (Stan)
- Reworked netdev locking to have consistent order (Stan, Kuba)
- Return 'not supported' in API patch (Stan)
- Improved ynl documentation (Kuba)
- Added 'max: s32-max' in ynl spec for ifindex (Kuba)
- Added also queue type in ynl to have user specify rx to make it obvious (Kuba)
- Use of netdev_hold (Kuba)
- Avoid static inlines from another header (Kuba)
- Squashed some commits (Kuba, Stan)
- Removed ndo_{peer,unpeer}_queues callback and simplified code (Kuba)
- Improved commit messages (Toke, Kuba, Stan, zf)
- Got rid of locking genl_sk_priv_get (Stan)
- Removed af_xdp cleanup churn (Maciej)
- Added netdev locking asserts (Stan)
- Reject ethtool ioctl path queue resizing (Kuba)
- Added kdoc for ndo_queue_create (Stan)
- Uninvert logic in netkit single dev mode (Jordan)
- Added binding support for multiple queues
Daniel Borkmann (10):
net: Add queue-create operation
net: Implement netdev_nl_queue_create_doit
net: Add lease info to queue-get response
net, ethtool: Disallow leased real rxqs to be resized
net: Slightly simplify net_mp_{open,close}_rxq
xsk: Extend xsk_rcv_check validation
xsk: Proxy pool management for leased queues
netkit: Add single device mode for netkit
netkit: Add netkit notifier to check for unregistering devices
netkit: Add xsk support for af_xdp applications
David Wei (6):
net: Proxy net_mp_{open,close}_rxq for leased queues
net: Proxy netdev_queue_get_dma_dev for leased queues
netkit: Implement rtnl_link_ops->alloc and ndo_queue_create selftests/net: Add bpf skb forwarding program selftests/net: Add env for container based tests selftests/net: Add netkit container tests
Documentation/netlink/specs/netdev.yaml | 44 +++ drivers/net/netkit.c | 360 +++++++++++++++--- include/linux/netdevice.h | 6 + include/net/netdev_queues.h | 19 +- include/net/netdev_rx_queue.h | 21 +- include/net/page_pool/memory_provider.h | 14 +- include/net/xdp_sock_drv.h | 2 +- include/uapi/linux/if_link.h | 6 + include/uapi/linux/netdev.h | 11 + io_uring/zcrx.c | 4 +- net/core/dev.c | 7 + net/core/dev.h | 2 + net/core/devmem.c | 6 +- net/core/netdev-genl-gen.c | 20 + net/core/netdev-genl-gen.h | 2 + net/core/netdev-genl.c | 208 ++++++++++ net/core/netdev_queues.c | 74 +++- net/core/netdev_rx_queue.c | 190 +++++++-- net/ethtool/channels.c | 12 +- net/ethtool/ioctl.c | 9 +- net/xdp/xsk.c | 79 +++- tools/include/uapi/linux/netdev.h | 11 + .../testing/selftests/drivers/net/README.rst | 7 + .../testing/selftests/drivers/net/hw/Makefile | 2 + .../drivers/net/hw/lib/py/__init__.py | 7 +- .../selftests/drivers/net/hw/nk_forward.bpf.c | 49 +++ .../selftests/drivers/net/hw/nk_netns.py | 23 ++ .../selftests/drivers/net/hw/nk_qlease.py | 90 +++++ .../selftests/drivers/net/lib/py/__init__.py | 7 +- .../selftests/drivers/net/lib/py/env.py | 120 ++++++ 30 files changed, 1272 insertions(+), 140 deletions(-) create mode 100644 tools/testing/selftests/drivers/net/hw/nk_forward.bpf.c create mode 100755 tools/testing/selftests/drivers/net/hw/nk_netns.py create mode 100755 tools/testing/selftests/drivers/net/hw/nk_qlease.py
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 1/16 net: Add queue-create operation
Add a ynl netdev family operation called queue-create that creates a new queue on a netdevice:
name: queue-create
attribute-set: queue
flags: [admin-perm]
do:
request:
attributes:
- ifindex
- type
- lease
reply: &queue-create-op
attributes:
- id
This is a generic operation such that it can be extended for various use cases in future. Right now it is mandatory to specify ifindex, the queue type which is enforced to rx and a lease. The newly created queue id is returned to the caller.
A queue from a virtual device can have a lease which refers to another queue from a physical device. This is useful for memory providers and AF_XDP operations which take an ifindex and queue id to allow applications to bind against virtual devices in containers. The lease couples both queues together and allows to proxy the operations from a virtual device in a container to the physical device.
In future, the nested lease attribute can be lifted and made optional for other use-cases such as dynamic queue creation for physical netdevs. The lack of lease and the specification of the physical device as an ifindex will imply that we need a real queue to be allocated. Similarly, the queue type enforcement to rx can then be lifted as well to support tx.
An early implementation had only driver-specific integration [0], but in order for other virtual devices to reuse, it makes sense to have this as a generic API in core net.
For leasing queues, the virtual netdev must have real_num_rx_queue less than num_rx_queues at the time of calling queue-create. The queue-type must be rx as only rx queues are supported for leasing for now. We also enforce that the queue-create ifindex must point to a virtual device, and that the nested lease attribute's ifindex must point to a physical device. The nested lease attribute set contains a netns-id attribute which is optional and can specify a netns-id relative to the caller's netns. It requires cap_net_admin and if the netns-id attribute is not specified, the lease ifindex will be retrieved from the current netns. Also, it is modeled as an s32 type similarly as done elsewhere in the stack.
--- Documentation/netlink/specs/netdev.yaml | 44 +++++++++++++++++++++++++ include/uapi/linux/netdev.h | 11 +++++++ net/core/netdev-genl-gen.c | 20 +++++++++++ net/core/netdev-genl-gen.h | 2 ++ net/core/netdev-genl.c | 5 +++ tools/include/uapi/linux/netdev.h | 11 +++++++ 6 files changed, 93 insertions(+)
Documentation/netlink/specs/netdev.yaml +44 -0
@@ -339,6 +339,15 @@ attribute-sets: doc: XSK information for this queue, if any. type: nest nested-attributes: xsk-info + - + name: lease + doc: | + A queue from a virtual device can have a lease which refers to + another queue from a physical device. This is useful for memory + providers and AF_XDP operations which take an ifindex and queue id + to allow applications to bind against virtual devices in containers. + type: nest + nested-attributes: lease - name: qstats doc: |
@@ -537,6 +546,24 @@ attribute-sets: name: id - name: type + - + name: lease + attributes: + - + name: ifindex + doc: The netdev ifindex to lease the queue from. + type: u32 + checks: + min: 1 + - + name: queue + doc: The netdev queue to lease from. + type: nest + nested-attributes: queue-id + - + name: netns-id + doc: The network namespace id of the netdev. + type: s32 - name: dmabuf attributes:
@@ -686,6 +713,7 @@ operations: - dmabuf - io-uring - xsk + - lease dump: request: attributes:
@@ -797,6 +825,22 @@ operations: reply: attributes: - id + - + name: queue-create + doc: | + Create a new queue for the given netdevice. Whether this operation + is supported depends on the device and the driver. + attribute-set: queue + flags: [admin-perm] + do: + request: + attributes: + - ifindex + - type + - lease + reply: &queue-create-op + attributes: + - id kernel-family: headers: ["net/netdev_netlink.h"]
include/uapi/linux/netdev.h +11 -0
@@ -160,6 +160,7 @@ enum {NETDEV_A_QUEUE_DMABUF ,NETDEV_A_QUEUE_IO_URING ,NETDEV_A_QUEUE_XSK , +NETDEV_A_QUEUE_LEASE , __NETDEV_A_QUEUE_MAX,NETDEV_A_QUEUE_MAX = (__NETDEV_A_QUEUE_MAX -1 )
@@ -202,6 +203,15 @@ enum {NETDEV_A_QSTATS_MAX = (__NETDEV_A_QSTATS_MAX -1 ) }; +enum { +NETDEV_A_LEASE_IFINDEX =1 , +NETDEV_A_LEASE_QUEUE , +NETDEV_A_LEASE_NETNS_ID , + + __NETDEV_A_LEASE_MAX, +NETDEV_A_LEASE_MAX = (__NETDEV_A_LEASE_MAX -1 ) +}; +enum {NETDEV_A_DMABUF_IFINDEX =1 ,NETDEV_A_DMABUF_QUEUES ,
@@ -228,6 +238,7 @@ enum {NETDEV_CMD_BIND_RX ,NETDEV_CMD_NAPI_SET ,NETDEV_CMD_BIND_TX , +NETDEV_CMD_QUEUE_CREATE , __NETDEV_CMD_MAX,NETDEV_CMD_MAX = (__NETDEV_CMD_MAX -1 )
net/core/netdev-genl-gen.c +20 -0
@@ -28,6 +28,12 @@ static const struct netlink_range_validation netdev_a_napi_defer_hard_irqs_range };/* Common nested types */ +const struct nla_policy netdev_lease_nl_policy[NETDEV_A_LEASE_NETNS_ID +1 ] = { + [NETDEV_A_LEASE_IFINDEX ] =NLA_POLICY_MIN (NLA_U32 ,1 ), + [NETDEV_A_LEASE_QUEUE ] =NLA_POLICY_NESTED (netdev_queue_id_nl_policy), + [NETDEV_A_LEASE_NETNS_ID ] = { .type =NLA_S32 , }, +}; +const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PAGE_POOL_IFINDEX +1 ] = { [NETDEV_A_PAGE_POOL_ID ] =NLA_POLICY_FULL_RANGE (NLA_UINT , &netdev_a_page_pool_id_range), [NETDEV_A_PAGE_POOL_IFINDEX ] =NLA_POLICY_FULL_RANGE (NLA_U32 , &netdev_a_page_pool_ifindex_range),
@@ -107,6 +113,13 @@ static const struct nla_policy netdev_bind_tx_nl_policy[NETDEV_A_DMABUF_FD + 1] [NETDEV_A_DMABUF_FD] = { .type = NLA_U32, }, }; +/* NETDEV_CMD_QUEUE_CREATE - do */ +static const struct nla_policy netdev_queue_create_nl_policy[NETDEV_A_QUEUE_LEASE +1 ] = { + [NETDEV_A_QUEUE_IFINDEX ] =NLA_POLICY_MIN (NLA_U32 ,1 ), + [NETDEV_A_QUEUE_TYPE ] =NLA_POLICY_MAX (NLA_U32 ,1 ), + [NETDEV_A_QUEUE_LEASE ] =NLA_POLICY_NESTED (netdev_lease_nl_policy), +}; +/* Ops table for netdev */ static const struct genl_split_ops netdev_nl_ops[] = { {
@@ -205,6 +218,13 @@ static const struct genl_split_ops netdev_nl_ops[] = { .maxattr = NETDEV_A_DMABUF_FD, .flags = GENL_CMD_CAP_DO, }, + { + .cmd = NETDEV_CMD_QUEUE_CREATE, + .doit = netdev_nl_queue_create_doit, + .policy = netdev_queue_create_nl_policy, + .maxattr = NETDEV_A_QUEUE_LEASE, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, };static const struct genl_multicast_group netdev_nl_mcgrps[] = {
net/core/netdev-genl-gen.h +2 -0
@@ -14,6 +14,7 @@#include <net/netdev_netlink.h> /* Common nested types */ +extern const struct nla_policy netdev_lease_nl_policy[NETDEV_A_LEASE_NETNS_ID +1 ];extern const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PAGE_POOL_IFINDEX +1 ];extern const struct nla_policy netdev_queue_id_nl_policy[NETDEV_A_QUEUE_TYPE +1 ];
@@ -36,6 +37,7 @@ int netdev_nl_qstats_get_dumpit(struct sk_buff *skb,int netdev_nl_bind_rx_doit (struct sk_buff *skb,struct genl_info *info);int netdev_nl_napi_set_doit (struct sk_buff *skb,struct genl_info *info);int netdev_nl_bind_tx_doit (struct sk_buff *skb,struct genl_info *info); +int netdev_nl_queue_create_doit (struct sk_buff *skb,struct genl_info *info);enum {NETDEV_NLGRP_MGMT ,
net/core/netdev-genl.c +5 -0
@@ -1120,6 +1120,11 @@ int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info)return err; } +int netdev_nl_queue_create_doit (struct sk_buff *skb,struct genl_info *info) +{ +return -EOPNOTSUPP ; +} +void netdev_nl_sock_priv_init (struct netdev_nl_sock *priv) {INIT_LIST_HEAD (&priv->bindings);
tools/include/uapi/linux/netdev.h +11 -0
@@ -160,6 +160,7 @@ enum {NETDEV_A_QUEUE_DMABUF ,NETDEV_A_QUEUE_IO_URING ,NETDEV_A_QUEUE_XSK , +NETDEV_A_QUEUE_LEASE , __NETDEV_A_QUEUE_MAX,NETDEV_A_QUEUE_MAX = (__NETDEV_A_QUEUE_MAX -1 )
@@ -202,6 +203,15 @@ enum {NETDEV_A_QSTATS_MAX = (__NETDEV_A_QSTATS_MAX -1 ) }; +enum { +NETDEV_A_LEASE_IFINDEX =1 , +NETDEV_A_LEASE_QUEUE , +NETDEV_A_LEASE_NETNS_ID , + + __NETDEV_A_LEASE_MAX, +NETDEV_A_LEASE_MAX = (__NETDEV_A_LEASE_MAX -1 ) +}; +enum {NETDEV_A_DMABUF_IFINDEX =1 ,NETDEV_A_DMABUF_QUEUES ,
@@ -228,6 +238,7 @@ enum {NETDEV_CMD_BIND_RX ,NETDEV_CMD_NAPI_SET ,NETDEV_CMD_BIND_TX , +NETDEV_CMD_QUEUE_CREATE , __NETDEV_CMD_MAX,NETDEV_CMD_MAX = (__NETDEV_CMD_MAX -1 )
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 2/16 net: Implement netdev_nl_queue_create_doit
In reply to Daniel BorkmannImplement netdev_nl_queue_create_doit which creates a new rx queue in a virtual netdev and then leases it to a rx queue in a physical netdev.
Example with ynl client:
# ./pyynl/cli.py \
--spec ~/netlink/specs/netdev.yaml \
--do queue-create \
--json '{"ifindex": 8, "type": "rx", "lease": {"ifindex": 4, "queue": {"type": "rx", "id": 15}}}'
{'id': 1}Note that the netdevice locking order is always from the virtual to the physical device.
--- include/net/netdev_queues.h | 19 +++- include/net/netdev_rx_queue.h | 9 +- include/net/xdp_sock_drv.h | 2 +- net/core/dev.c | 7 ++ net/core/dev.h | 2 + net/core/netdev-genl.c | 169 +++++++++++++++++++++++++++++++++- net/core/netdev_queues.c | 57 ++++++++++++ net/core/netdev_rx_queue.c | 46 ++++++++- net/xdp/xsk.c | 2 +- 9 files changed, 301 insertions(+), 12 deletions(-)
include/net/netdev_queues.h +16 -3
@@ -150,6 +150,11 @@ enum {* When NIC-wide config is changed the callback will * be invoked for all queues. * +* @ndo_queue_create: Create a new RX queue which can be leased to another queue. +* Ops on this queue are redirected to the leased queue e.g. +* when opening a memory provider. Return the new queue id on +* success. Return negative error code on failure. +* * @supported_params: Bitmask of supported parameters, see QCFG_*. * * Note that @ndo_queue_mem_alloc and @ndo_queue_mem_free may be called while
@@ -178,6 +183,7 @@ struct netdev_queue_mgmt_ops { struct netlink_ext_ack *extack);struct device * (*ndo_queue_get_dma_dev)(struct net_device *dev,int idx); +int (*ndo_queue_create)(struct net_device *dev);unsigned int supported_params; };
@@ -185,7 +191,9 @@ struct netdev_queue_mgmt_ops {void netdev_queue_config (struct net_device *dev,int rxq,struct netdev_queue_config *qcfg); -bool netif_rxq_has_unreadable_mp (struct net_device *dev,int idx); +bool netif_rxq_has_unreadable_mp (struct net_device *dev,unsigned int rxq_idx); +bool netif_rxq_has_mp (struct net_device *dev,unsigned int rxq_idx); +bool netif_rxq_is_leased (struct net_device *dev,unsigned int rxq_idx);/** * DOC: Lockless queue stopping / waking helpers.
@@ -374,5 +382,10 @@ static inline unsigned int netif_xmit_timeout_ms(struct netdev_queue *txq) })struct device *netdev_queue_get_dma_dev (struct net_device *dev,int idx); - -#endif +bool netdev_can_create_queue (const struct net_device *dev, +struct netlink_ext_ack *extack); +bool netdev_can_lease_queue (const struct net_device *dev, +struct netlink_ext_ack *extack); +bool netdev_queue_busy (struct net_device *dev,int idx, +struct netlink_ext_ack *extack); +#endif /* _LINUX_NET_QUEUES_H */
include/net/netdev_rx_queue.h +7 -2
@@ -30,6 +30,8 @@ struct netdev_rx_queue {struct napi_struct *napi;struct netdev_queue_config qcfg;struct pp_memory_provider_params mp_params; +struct netdev_rx_queue *lease; +netdevice_tracker lease_tracker; }____cacheline_aligned_in_smp ;/*
@@ -59,5 +61,8 @@ get_netdev_rx_queue_index(struct netdev_rx_queue *queue) }int netdev_rx_queue_restart (struct net_device *dev,unsigned int rxq); - -#endif +void netdev_rx_queue_lease (struct netdev_rx_queue *rxq_dst, +struct netdev_rx_queue *rxq_src); +void netdev_rx_queue_unlease (struct netdev_rx_queue *rxq_dst, +struct netdev_rx_queue *rxq_src); +#endif /* _LINUX_NETDEV_RX_QUEUE_H */
include/net/xdp_sock_drv.h +1 -1
@@ -28,7 +28,7 @@ void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries);bool xsk_tx_peek_desc (struct xsk_buff_pool *pool,struct xdp_desc *desc);u32 xsk_tx_peek_release_desc_batch (struct xsk_buff_pool *pool,u32 max);void xsk_tx_release (struct xsk_buff_pool *pool); -struct xsk_buff_pool *xsk_get_pool_from_qid (struct net_device *dev, +struct xsk_buff_pool *xsk_get_pool_from_qid (const struct net_device *dev,u16 queue_id);void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool);void xsk_set_tx_need_wakeup (struct xsk_buff_pool *pool);
net/core/dev.c +7 -0
@@ -1114,6 +1114,13 @@ netdev_get_by_index_lock_ops_compat(struct net *net, int ifindex)return __netdev_put_lock_ops_compat (dev, net); } +struct net_device * +netdev_put_lock (struct net_device *dev,netdevice_tracker *tracker) +{ +netdev_tracker_free (dev, tracker); +return __netdev_put_lock (dev,dev_net (dev)); +} +struct net_device *netdev_xa_find_lock (struct net *net,struct net_device *dev,unsigned long *index)
net/core/dev.h +2 -0
@@ -31,6 +31,8 @@ netdev_napi_by_id_lock(struct net *net, unsigned int napi_id);struct net_device *dev_get_by_napi_id (unsigned int napi_id);struct net_device *__netdev_put_lock (struct net_device *dev,struct net *net); +struct net_device *netdev_put_lock (struct net_device *dev, +netdevice_tracker *tracker);struct net_device *netdev_xa_find_lock (struct net *net,struct net_device *dev,unsigned long *index);
net/core/netdev-genl.c +168 -1
175 lines not on this page · raw message
net/core/netdev_queues.c +57 -0
66 lines not on this page · raw message
net/core/netdev_rx_queue.c +42 -4
57 lines not on this page · raw message
net/xdp/xsk.c +1 -1
@@ -103,7 +103,7 @@ bool xsk_uses_need_wakeup(struct xsk_buff_pool *pool) }EXPORT_SYMBOL (xsk_uses_need_wakeup); -struct xsk_buff_pool *xsk_get_pool_from_qid (struct net_device *dev, +struct xsk_buff_pool *xsk_get_pool_from_qid (const struct net_device *dev,u16 queue_id) {if (queue_id < dev->real_num_rx_queues)
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 02/16] net: Implement netdev_nl_queue_create_doit
Quoted 7 lines · On Thu, 29 Jan 2026 23:28:16 +0100 Daniel Borkmann wrote:
Implement
netdev_nl_queue_create_doitwhich creates a new rx queue in a virtual netdev and then leases it to a rx queue in a physical netdev.Example with ynl client:
# ./pyynl/cli.py \ --spec ~/netlink/specs/netdev.yaml \
nit: please use "ynl --family netdev" instead of ./pyynl/cli.py --spec...
Quoted 2 lines
Note that the netdevice locking order is always from the virtual to the physical device.
There is a big comment in netdevice.h (search for "netdev-scope lock") documenting the instance lock, since you are adding technical deb^w^w an ordering rule we should mention it there concisely and also update the "netdev instance lock" section of netdevices.rst
include/net/netdev_queues.h struct netdev_queue_mgmt_ops { · line 186
Quoted 1 line
include/net/netdev_queues.h +1
-0+int (*ndo_queue_create)(struct net_device *dev);
please propagate extack to the driver
include/net/netdev_queues.h struct netdev_queue_mgmt_ops { · line 196
Quoted 10 lines
include/net/netdev_queues.h +3
-1unsigned int supported_params; };@@ -185,7 +191,9 @@ struct netdev_queue_mgmt_ops {void netdev_queue_config (struct net_device *dev,int rxq,struct netdev_queue_config *qcfg); -bool netif_rxq_has_unreadable_mp (struct net_device *dev,int idx); +bool netif_rxq_has_unreadable_mp (struct net_device *dev,unsigned int rxq_idx); +bool netif_rxq_has_mp (struct net_device *dev,unsigned int rxq_idx); +bool netif_rxq_is_leased (struct net_device *dev,unsigned int rxq_idx);
The new function is internal to the core, please add the declarations in the appropriate place in net/core/dev.h instead.
Coincidentally we should probably delete netif_rxq_has_unreadable_mp() completely and pass the "has unreadable mp" as a flag inside qcfg instead. I'll clean that up after your patches.
Quoted 4 lines
include/net/netdev_queues.h +0
-0/** * DOC: Lockless queue stopping / waking helpers. @@ -374,5 +382,10 @@ static inline unsigned int netif_xmit_timeout_ms(struct netdev_queue *txq) })
include/net/netdev_queues.h static inline unsigned int netif_xmit_timeout_ms(struct netdev_queue *txq) · line 390
Quoted 10 lines
include/net/netdev_queues.h +6
-2struct device *netdev_queue_get_dma_dev (struct net_device *dev,int idx); - -#endif +bool netdev_can_create_queue (const struct net_device *dev, +struct netlink_ext_ack *extack); +bool netdev_can_lease_queue (const struct net_device *dev, +struct netlink_ext_ack *extack); +bool netdev_queue_busy (struct net_device *dev,int idx, +struct netlink_ext_ack *extack);
ditto. FWIW it's fine for ethtool to include using a relative path, it already does that in a few places
include/net/netdev_rx_queue.h struct netdev_rx_queue { · line 32
Quoted 9 lines
include/net/netdev_queues.h +1
-0+#endif /* _LINUX_NET_QUEUES_H */ include/net/netdev_rx_queue.h +0
-0@@ -30,6 +30,8 @@ struct netdev_rx_queue {struct napi_struct *napi;struct netdev_queue_config qcfg;struct pp_memory_provider_params mp_params;
Could you add a comment here explaining whether this pointer is to or from the lease or both, depending whether the device is virt?
Quoted 6 lines
include/net/netdev_rx_queue.h +2
-0+struct netdev_rx_queue *lease; +netdevice_tracker lease_tracker; }____cacheline_aligned_in_smp ;/*
include/net/netdev_rx_queue.h get_netdev_rx_queue_index(struct netdev_rx_queue *queue) · line 67
Quoted 9 lines
include/net/netdev_rx_queue.h +4
-2@@ -59,5 +61,8 @@ get_netdev_rx_queue_index(struct netdev_rx_queue *queue) }int netdev_rx_queue_restart (struct net_device *dev,unsigned int rxq); - -#endif +void netdev_rx_queue_lease (struct netdev_rx_queue *rxq_dst, +struct netdev_rx_queue *rxq_src); +void netdev_rx_queue_unlease (struct netdev_rx_queue *rxq_dst, +struct netdev_rx_queue *rxq_src);
dev.h
Quoted 30 lines
include/net/netdev_rx_queue.h +27
-1int netdev_nl_queue_create_doit (struct sk_buff *skb,struct genl_info *info) { -return -EOPNOTSUPP ; +const int qmaxtype =ARRAY_SIZE (netdev_queue_id_nl_policy) -1 ; +const int lmaxtype =ARRAY_SIZE (netdev_lease_nl_policy) -1 ; +int err, ifindex, ifindex_lease, queue_id, queue_id_lease; +struct nlattr *qtb[ARRAY_SIZE (netdev_queue_id_nl_policy)]; +struct nlattr *ltb[ARRAY_SIZE (netdev_lease_nl_policy)]; +struct netdev_rx_queue *rxq, *rxq_lease; +struct net_device *dev, *dev_lease; +netdevice_tracker dev_tracker; +s32 netns_lease =-1 ; +struct nlattr *nest; +struct sk_buff *rsp; +struct net *net; +void *hdr; + +if (GENL_REQ_ATTR_CHECK (info,NETDEV_A_QUEUE_IFINDEX ) || +GENL_REQ_ATTR_CHECK (info,NETDEV_A_QUEUE_TYPE ) || +GENL_REQ_ATTR_CHECK (info,NETDEV_A_QUEUE_LEASE )) +return -EINVAL ; +if (nla_get_u32 (info->attrs[NETDEV_A_QUEUE_TYPE ]) != +NETDEV_QUEUE_TYPE_RX ) { +NL_SET_BAD_ATTR (info->extack, info->attrs[NETDEV_A_QUEUE_TYPE ]); +return -EINVAL ; + } + + ifindex =nla_get_u32 (info->attrs[NETDEV_A_QUEUE_IFINDEX ]); + + nest = info->attrs[NETDEV_A_QUEUE_LEASE ];
include/net/netdev_rx_queue.h get_netdev_rx_queue_index(struct netdev_rx_queue *queue)
Quoted 10 lines
include/net/netdev_rx_queue.h +10
-0+ err =nla_parse_nested (ltb, lmaxtype, nest, + netdev_lease_nl_policy, info->extack); +if (err <0 ) +return err; +if (NL_REQ_ATTR_CHECK (info->extack, nest, ltb,NETDEV_A_LEASE_IFINDEX ) || +NL_REQ_ATTR_CHECK (info->extack, nest, ltb,NETDEV_A_LEASE_QUEUE )) +return -EINVAL ; +if (ltb[NETDEV_A_LEASE_NETNS_ID ]) { + netns_lease =nla_get_s32 (ltb[NETDEV_A_LEASE_NETNS_ID ]); +if (netns_lease <0 ) {
Let's add this to the spec / policy then?
checks:
min: 0
Quoted 48 lines
include/net/netdev_rx_queue.h +48
-0+NL_SET_BAD_ATTR (info->extack, ltb[NETDEV_A_LEASE_NETNS_ID ]); +return -EINVAL ; + } +if (!capable (CAP_NET_ADMIN )) +return -EPERM ; + } + + ifindex_lease =nla_get_u32 (ltb[NETDEV_A_LEASE_IFINDEX ]); + + nest = ltb[NETDEV_A_LEASE_QUEUE ]; + err =nla_parse_nested (qtb, qmaxtype, nest, + netdev_queue_id_nl_policy, info->extack); +if (err <0 ) +return err; +if (NL_REQ_ATTR_CHECK (info->extack, nest, qtb,NETDEV_A_QUEUE_ID ) || +NL_REQ_ATTR_CHECK (info->extack, nest, qtb,NETDEV_A_QUEUE_TYPE )) +return -EINVAL ; +if (nla_get_u32 (qtb[NETDEV_A_QUEUE_TYPE ]) !=NETDEV_QUEUE_TYPE_RX ) { +NL_SET_BAD_ATTR (info->extack, qtb[NETDEV_A_QUEUE_TYPE ]); +return -EINVAL ; + } + + queue_id_lease =nla_get_u32 (qtb[NETDEV_A_QUEUE_ID ]); + + rsp =genlmsg_new (GENLMSG_DEFAULT_SIZE ,GFP_KERNEL ); +if (!rsp) +return -ENOMEM ; + + hdr =genlmsg_iput (rsp, info); +if (!hdr) { + err = -EMSGSIZE ; + goto err_genlmsg_free; + } + +/* Locking order is always from the virtual to the physical device +* since this is also the same order when applications open the +* memory provider later on. +*/ + dev =netdev_get_by_index_lock (genl_info_net (info), ifindex); +if (!dev) { + err = -ENODEV ; + goto err_genlmsg_free; + } +if (!netdev_can_create_queue (dev, info->extack)) { + err = -EINVAL ; + goto err_unlock_dev; + } +
include/net/netdev_rx_queue.h
Quoted 10 lines
include/net/netdev_rx_queue.h +10
-0+ net =genl_info_net (info); +if (netns_lease >=0 ) { + net =get_net_ns_by_id (net, netns_lease); +if (!net) { + err = -ENONET ; + goto err_unlock_dev; + } + } +if (net_eq (net,dev_net (dev)) && + ifindex == ifindex_lease) {
Is this check actually needed? The device can't be physical and virtual at once so the locking safety check would fail anyway, no?
Quoted 27 lines
include/net/netdev_rx_queue.h +27
-0+NL_SET_ERR_MSG (info->extack, +"Lease ifindex cannot be the same as queue creation ifindex" ); + err = -EINVAL ; + goto err_put_netns; + } + + dev_lease =netdev_get_by_index (net, ifindex_lease, &dev_tracker, +GFP_KERNEL ); +if (!dev_lease) { + err = -ENODEV ; + goto err_put_netns; + } +if (!netdev_can_lease_queue (dev_lease, info->extack)) { +netdev_put (dev_lease, &dev_tracker); + err = -EINVAL ; + goto err_put_netns; + } + + dev_lease =netdev_put_lock (dev_lease, &dev_tracker); +if (!dev_lease) { + err = -ENODEV ; + goto err_put_netns; + } +if (queue_id_lease >= dev_lease->real_num_rx_queues) { + err = -ERANGE ; +NL_SET_BAD_ATTR (info->extack, qtb[NETDEV_A_QUEUE_ID ]); + goto err_unlock_dev_lease;
include/net/netdev_rx_queue.h
Quoted 10 lines
include/net/netdev_rx_queue.h +10
-0+ } +if (netdev_queue_busy (dev_lease, queue_id_lease, info->extack)) { + err = -EBUSY ; + goto err_unlock_dev_lease; + } + + rxq_lease =__netif_get_rx_queue (dev_lease, queue_id_lease); + rxq =__netif_get_rx_queue (dev, dev->real_num_rx_queues -1 ); + +if (rxq->lease && rxq->lease->dev != dev_lease) {
IIUC the simplification of having all leases from one devices is still a netkit thing? I mean - there's nothing in the core that depends on this, just the cleanup / notifier handling in netkit? If that's the case let's move this check into netkit.
Sorry if you moved this here because I asked to move as much as possible into the core.
include/net/netdev_rx_queue.h
Quoted 10 lines
include/net/netdev_rx_queue.h +10
-0+ err = -EOPNOTSUPP ; +NL_SET_ERR_MSG (info->extack, +"Leasing multiple queues from different devices not supported" ); + goto err_unlock_dev_lease; + } + + err = queue_id = dev->queue_mgmt_ops->ndo_queue_create (dev); +if (err <0 ) { +NL_SET_ERR_MSG (info->extack, +"Device is unable to create a new queue" );
As flagged, if we pass extack to the driver it should be able to give us a more accurate reason
include/net/netdev_rx_queue.h
Quoted 5 lines
include/net/netdev_rx_queue.h +5
-0+ goto err_unlock_dev_lease; + } + + rxq =__netif_get_rx_queue (dev, queue_id); +netdev_rx_queue_lease (rxq, rxq_lease);
nit: the call to __netif_get_rx_queue() could move into netdev_rx_queue_lease()
Quoted 7 lines
include/net/netdev_rx_queue.h +7
-0+nla_put_u32 (rsp,NETDEV_A_QUEUE_ID , queue_id); +genlmsg_end (rsp, hdr); + +netdev_unlock (dev_lease); +netdev_unlock (dev); +if (netns_lease >=0 ) +put_net (net);
Daniel Borkmann
maintainerRe: [PATCH net-next v8 02/16] net: Implement netdev_nl_queue_create_doit
On 2/1/26 12:45 AM, Jakub Kicinski wrote:
[...]
Quoted 5 lines
net/core/netdev-genl.c int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) · line 1219
Quoted 2 lines
net/core/netdev-genl.c +2
-0+if (net_eq (net,dev_net (dev)) && + ifindex == ifindex_lease) {Is this check actually needed? The device can't be physical and virtual at once so the locking safety check would fail anyway, no?
It would fail, correct. So I've dropped it locally for v9.
[...]
Quoted 15 lines
net/core/netdev-genl.c int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) · line 1256
Quoted 9 lines
net/core/netdev-genl.c +9
-0+if (netdev_queue_busy (dev_lease, queue_id_lease, info->extack)) { + err = -EBUSY ; + goto err_unlock_dev_lease; + } + + rxq_lease =__netif_get_rx_queue (dev_lease, queue_id_lease); + rxq =__netif_get_rx_queue (dev, dev->real_num_rx_queues -1 ); + +if (rxq->lease && rxq->lease->dev != dev_lease) {IIUC the simplification of having all leases from one devices is still a netkit thing? I mean - there's nothing in the core that depends on this, just the cleanup / notifier handling in netkit? If that's the case let's move this check into netkit.
Quoted 2 lines
Sorry if you moved this here because I asked to move as much as possible into the core.
Imho it was a good suggestion to move it here and I would prefer to leave this in the core, and only really move it into a per-driver constraint once we see a driver wanting to support such setting. Otherwise we would propagate AF_XDP support on the device where some queues have support but others don't.
Thanks,
Daniel
Jakub Kicinski
maintainerRe: [PATCH net-next v8 02/16] net: Implement netdev_nl_queue_create_doit
Quoted 10 lines · On Thu, 5 Mar 2026 05:38:06 +0100 Daniel Borkmann wrote:
Quoted 10 lines
net/core/netdev-genl.c int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) · line 1256
Quoted 4 lines
net/core/netdev-genl.c +4
-0+ rxq_lease =__netif_get_rx_queue (dev_lease, queue_id_lease); + rxq =__netif_get_rx_queue (dev, dev->real_num_rx_queues -1 ); + +if (rxq->lease && rxq->lease->dev != dev_lease) {IIUC the simplification of having all leases from one devices is still a netkit thing? I mean - there's nothing in the core that depends on this, just the cleanup / notifier handling in netkit? If that's the case let's move this check into netkit.
Quoted 7 lines
Quoted 2 lines
Sorry if you moved this here because I asked to move as much as possible into the core.
Imho it was a good suggestion to move it here and I would prefer to leave this in the core, and only really move it into a per-driver constraint once we see a driver wanting to support such setting. Otherwise we would propagate
AF_XDPsupport on the device where some queues have support but others don't.
Alright, let's add a comment explaining the status and plan then?
Daniel Borkmann
maintainerRe: [PATCH net-next v8 02/16] net: Implement netdev_nl_queue_create_doit
Quoted 13 lines · Jakub Kicinski
Quoted 12 lines · On Thu, 5 Mar 2026 05:38:06 +0100 Daniel Borkmann wrote:
Quoted 12 lines
net/core/netdev-genl.c int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) · line 1256
Quoted 4 lines
net/core/netdev-genl.c +4
-0+ rxq_lease =__netif_get_rx_queue (dev_lease, queue_id_lease); + rxq =__netif_get_rx_queue (dev, dev->real_num_rx_queues -1 ); + +if (rxq->lease && rxq->lease->dev != dev_lease) {IIUC the simplification of having all leases from one devices is still a netkit thing? I mean - there's nothing in the core that depends on this, just the cleanup / notifier handling in netkit? If that's the case let's move this check into netkit.
Sorry if you moved this here because I asked to move as much as possible into the core.
Quoted 7 lines
Quoted 5 lines
Imho it was a good suggestion to move it here and I would prefer to leave this in the core, and only really move it into a per-driver constraint once we see a driver wanting to support such setting. Otherwise we would propagate
AF_XDPsupport on the device where some queues have support but others don't.Alright, let's add a comment explaining the status and plan then?
Yep, will add.
Daniel Borkmann
maintainerPATCH v8 3/16 net: Add lease info to queue-get response
In reply to Daniel BorkmannPopulate nested lease info to the queue-get response that returns the ifindex, queue id with type and optionally netns id if the device resides in a different netns.
Example with ynl client:
# ip a
[...]
4: enp10s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:24 qdisc mq state UP group default qlen 1000
link/ether e8:eb:d3:a3:43:f6 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 scope global enp10s0f0np0
valid_lft forever preferred_lft forever
inet6 fe80::eaeb:d3ff:fea3:43f6/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
[...]# ethtool -i enp10s0f0np0 driver: mlx5_core [...]
# ./pyynl/cli.py \
--spec ~/netlink/specs/netdev.yaml \
--do queue-get \
--json '{"ifindex": 4, "id": 15, "type": "rx"}'
{'id': 15,
'ifindex': 4,
'lease': {'ifindex': 8, 'netns-id': 0, 'queue': {'id': 1, 'type': 'rx'}},
'napi-id': 8227,
'type': 'rx',
'xsk': {}}# ip netns list foo (id: 0)
# ip netns exec foo ip a
[...]
8: nk@NONE: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::200:ff:fe00:0/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
[...]# ip netns exec foo ethtool -i nk driver: netkit [...]
# ip netns exec foo ls /sys/class/net/nk/queues/ rx-0 rx-1 tx-0
# ip netns exec foo ./pyynl/cli.py \
--spec ~/netlink/specs/netdev.yaml \
--do queue-get \
--json '{"ifindex": 8, "id": 1, "type": "rx"}'
{'id': 1, 'ifindex': 8, 'type': 'rx'}Note that the caller of netdev_nl_queue_fill_one() holds the netdevice lock. For the queue-get we do not lock both devices. When queues get {un,}leased, both devices are locked, thus if __netif_get_rx_queue_peer() returns true, the lease pointer points to a valid device. The netns-id is fetched via peernet2id_alloc() similarly as done in OVS.
--- include/net/netdev_rx_queue.h | 10 ++++++++ net/core/netdev-genl.c | 36 ++++++++++++++++++++++++++++ net/core/netdev_rx_queue.c | 45 +++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+)
include/net/netdev_rx_queue.h +10 -0
@@ -65,4 +65,14 @@ void netdev_rx_queue_lease(struct netdev_rx_queue *rxq_dst, struct netdev_rx_queue *rxq_src);void netdev_rx_queue_unlease (struct netdev_rx_queue *rxq_dst,struct netdev_rx_queue *rxq_src); +bool netif_rx_queue_lease_get_owner (struct net_device **dev,unsigned int *rxq); + +enum netif_lease_dir { +NETIF_VIRT_TO_PHYS , +NETIF_PHYS_TO_VIRT , +}; + +struct netdev_rx_queue * +__netif_get_rx_queue_lease (struct net_device **dev,unsigned int *rxq, +enum netif_lease_dir dir);#endif /* _LINUX_NETDEV_RX_QUEUE_H */
net/core/netdev-genl.c +36 -0
@@ -391,8 +391,11 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev, u32 q_idx, u32 q_type, const struct genl_info *info) { struct pp_memory_provider_params *params; +struct net_device *orig_netdev = netdev; +struct nlattr *nest_lease, *nest_queue;struct netdev_rx_queue *rxq;struct netdev_queue *txq; +u32 lease_q_idx = q_idx;void *hdr; hdr =genlmsg_iput (rsp, info);
@@ -410,6 +413,37 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev,if (nla_put_napi_id (rsp, rxq->napi)) goto nla_put_failure; +if (netif_rx_queue_lease_get_owner (&netdev, &lease_q_idx)) { +struct net *net, *peer_net; + + nest_lease =nla_nest_start (rsp,NETDEV_A_QUEUE_LEASE ); +if (!nest_lease) + goto nla_put_failure; + nest_queue =nla_nest_start (rsp,NETDEV_A_LEASE_QUEUE ); +if (!nest_queue) + goto nla_put_failure; +if (nla_put_u32 (rsp,NETDEV_A_QUEUE_ID , lease_q_idx)) + goto nla_put_failure; +if (nla_put_u32 (rsp,NETDEV_A_QUEUE_TYPE , q_type)) + goto nla_put_failure; +nla_nest_end (rsp, nest_queue); +if (nla_put_u32 (rsp,NETDEV_A_LEASE_IFINDEX , +READ_ONCE (netdev->ifindex))) + goto nla_put_failure; +rcu_read_lock (); + peer_net =dev_net_rcu (netdev); + net =dev_net_rcu (orig_netdev); +if (!net_eq (net, peer_net)) { +s32 id =peernet2id_alloc (net, peer_net,GFP_ATOMIC ); + +if (nla_put_s32 (rsp,NETDEV_A_LEASE_NETNS_ID , id)) + goto nla_put_failure_unlock; + } +rcu_read_unlock (); +nla_nest_end (rsp, nest_lease); + netdev = orig_netdev; + } + params = &rxq->mp_params;if (params->mp_ops && params->mp_ops->nl_fill (params->mp_priv, rsp, rxq))
@@ -437,6 +471,8 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev,return 0 ; +nla_put_failure_unlock: +rcu_read_unlock (); nla_put_failure:genlmsg_cancel (rsp, hdr);return -EMSGSIZE ;
net/core/netdev_rx_queue.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 03/16] net: Add lease info to queue-get response
include/net/netdev_rx_queue.h void netdev_rx_queue_lease(struct netdev_rx_queue *rxq_dst, · line 77
Quoted 12 lines · On Thu, 29 Jan 2026 23:28:17 +0100 Daniel Borkmann wrote:
Populate nested lease info to the queue-get response that returns the ifindex, queue id with type and optionally netns id if the device resides in a different netns.
include/net/netdev_rx_queue.h +8
-0+enum netif_lease_dir { +NETIF_VIRT_TO_PHYS , +NETIF_PHYS_TO_VIRT , +}; + +struct netdev_rx_queue * +__netif_get_rx_queue_lease (struct net_device **dev,unsigned int *rxq, +enum netif_lease_dir dir);
dev.h
Quoted 6 lines
#endif /*
_LINUX_NETDEV_RX_QUEUE_H*/net/core/netdev-genl.c +0
-0
net/core/netdev-genl.c netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev, · line 398
Quoted 8 lines
net/core/netdev-genl.c +3
-0@@ -391,8 +391,11 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev, u32 q_idx, u32 q_type, const struct genl_info *info) { struct pp_memory_provider_params *params; +struct net_device *orig_netdev = netdev; +struct nlattr *nest_lease, *nest_queue;struct netdev_rx_queue *rxq;struct netdev_queue *txq; +u32 lease_q_idx = q_idx;
No need to init this, no?
net/core/netdev-genl.c netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev, · line 416
Quoted 8 lines
net/core/netdev-genl.c +1
-0void *hdr; hdr =genlmsg_iput (rsp, info);@@ -410,6 +413,37 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev,if (nla_put_napi_id (rsp, rxq->napi)) goto nla_put_failure; +if (netif_rx_queue_lease_get_owner (&netdev, &lease_q_idx)) {
Does it work if we factored this out to a helper?
Also apparently the words lessee and lessor are legit English words.
I wonder if using those would make the code quite a bit easier to follow when we have to refer to the lease in a specific direction.
Jakub Kicinski
maintainerRe: [PATCH net-next v8 03/16] net: Add lease info to queue-get response
In reply to Daniel BorkmannQuoted 8 lines · On Thu, 29 Jan 2026 23:28:17 +0100 Daniel Borkmann wrote:
# ip netns exec foo ls /sys/class/net/nk/queues/ rx-0 rx-1 tx-0# ip netns exec foo ./pyynl/cli.py \ --spec ~/netlink/specs/netdev.yaml \ --do queue-get \ --json '{"ifindex": 8, "id": 1, "type": "rx"}' {'id': 1, 'ifindex': 8, 'type': 'rx'}
I think the memory provider is going to be missing here? We want the container to see its memory provider but IIUC the mp_priv ends up on the lessor's queue.
I started wondering if we don't need extra checks for XDP (MPs and XDP don't currently mix) but IIUC the mp ends up on the "real" queue, so XDP should be fine. So either we need to follow down here, or pop the mp pointers onto both queues?
Daniel Borkmann
maintainerPATCH v8 4/16 net, ethtool: Disallow leased real rxqs to be resized
In reply to Daniel BorkmannSimilar to AF_XDP, do not allow queues in a physical netdev to be resized by ethtool -L when they are leased.
--- net/ethtool/channels.c | 12 +++++++----- net/ethtool/ioctl.c | 9 +++++---- 2 files changed, 12 insertions(+), 9 deletions(-)
net/ethtool/channels.c +7 -5
@@ -1,6 +1,6 @@// SPDX-License-Identifier: GPL-2.0-only -#include <net/xdp_sock_drv.h> +#include <net/netdev_queues.h> #include "netlink.h" #include "common.h"
@@ -169,14 +169,16 @@ ethnl_set_channels(struct ethnl_req_info *req_info, struct genl_info *info)if (ret)return ret; -/* Disabling channels, query zero-copy AF_XDP sockets */ +/* ensure channels are not busy at the moment */ from_channel = channels.combined_count +min (channels.rx_count, channels.tx_count); -for (i = from_channel; i < old_total; i++) -if (xsk_get_pool_from_qid (dev, i)) { -GENL_SET_ERR_MSG (info,"requested channel counts are too low for existing zerocopy AF_XDP sockets" ); +for (i = from_channel; i < old_total; i++) { +if (netdev_queue_busy (dev, i,NULL )) { +GENL_SET_ERR_MSG (info, +"requested channel counts are too low due to busy queues (AF_XDP or queue leasing)" );return -EINVAL ; } + } ret = dev->ethtool_ops->set_channels (dev, &channels);return ret <0 ? ret :1 ;
net/ethtool/ioctl.c +5 -4
@@ -27,12 +27,13 @@#include <linux/net.h> #include <linux/pm_runtime.h> #include <linux/utsname.h> +#include <linux/ethtool_netlink.h> #include <net/devlink.h> #include <net/ipv6.h> -#include <net/xdp_sock_drv.h> #include <net/flow_offload.h> #include <net/netdev_lock.h> -#include <linux/ethtool_netlink.h> +#include <net/netdev_queues.h> +#include "common.h" /* State held across locks and calls for commands which have devlink fallback */
@@ -2282,12 +2283,12 @@ static noinline_for_stack int ethtool_set_channels(struct net_device *dev,if (ret)return ret; -/* Disabling channels, query zero-copy AF_XDP sockets */ +/* Disabling channels, query busy queues (AF_XDP, queue leasing) */ from_channel = channels.combined_count +min (channels.rx_count, channels.tx_count); to_channel = curr.combined_count +max (curr.rx_count, curr.tx_count);for (i = from_channel; i < to_channel; i++) -if (xsk_get_pool_from_qid (dev, i)) +if (netdev_queue_busy (dev, i,NULL ))return -EINVAL ; ret = dev->ethtool_ops->set_channels (dev, &channels);
Signature · 1 line
2.43.0
Re: [PATCH net-next v8 04/16] net, ethtool: Disallow leased real rxqs to be resized
Quoted 2 lines · On Thu, 29 Jan 2026 23:28:18 +0100 Daniel Borkmann wrote:
Similar to
AF_XDP, do not allow queues in a physical netdev to be resized by ethtool -L when they are leased.
I'd be tempted to factor the addition of netdev_queue_busy() out and get it merged quickly so that we can make this series smaller and more focused.
Daniel Borkmann
maintainerPATCH v8 5/16 net: Slightly simplify net_mp_{open,close}_rxq
In reply to Daniel Borkmannnet_mp_open_rxq is currently not used in the tree as all callers are using __net_mp_open_rxq directly, and net_mp_close_rxq is only used once while all other locations use __net_mp_close_rxq. Simplify this into a single API into __net_mp_{open,close}_rxq and rename the latter into net_mp_{open,close}_rxq given the __ prefix then becomes unnecessary.
--- include/net/page_pool/memory_provider.h | 14 +++++------- io_uring/zcrx.c | 4 +++- net/core/devmem.c | 6 ++--- net/core/netdev_rx_queue.c | 29 +++++-------------------- 4 files changed, 16 insertions(+), 37 deletions(-)
include/net/page_pool/memory_provider.h +5 -9
@@ -23,15 +23,11 @@ bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr);void net_mp_niov_set_page_pool (struct page_pool *pool,struct net_iov *niov);void net_mp_niov_clear_page_pool (struct net_iov *niov); -int net_mp_open_rxq (struct net_device *dev,unsigned ifq_idx, -struct pp_memory_provider_params *p); -int __net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, -const struct pp_memory_provider_params *p, -struct netlink_ext_ack *extack); -void net_mp_close_rxq (struct net_device *dev,unsigned ifq_idx, -struct pp_memory_provider_params *old_p); -void __net_mp_close_rxq (struct net_device *dev,unsigned int rxq_idx, -const struct pp_memory_provider_params *old_p); +int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack); +void net_mp_close_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *old_p);/** * net_mp_netmem_place_in_cache() - give a netmem to a page pool
io_uring/zcrx.c +3 -1
@@ -521,7 +521,9 @@ static void io_close_queue(struct io_zcrx_ifq *ifq) }if (netdev) { +netdev_lock (netdev);net_mp_close_rxq (netdev, ifq->if_rxq, &p); +netdev_unlock (netdev);netdev_put (netdev, &netdev_tracker); } ifq->if_rxq =-1 ;
@@ -806,7 +808,7 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, mp_param.mp_ops = &io_uring_pp_zc_ops; mp_param.mp_priv = ifq; - ret =__net_mp_open_rxq (ifq->netdev, reg.if_rxq, &mp_param,NULL ); + ret =net_mp_open_rxq (ifq->netdev, reg.if_rxq, &mp_param,NULL );if (ret) goto netdev_put_unlock;netdev_unlock (ifq->netdev);
net/core/devmem.c +3 -3
@@ -145,7 +145,7 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) rxq_idx =get_netdev_rx_queue_index (rxq); -__net_mp_close_rxq (binding->dev, rxq_idx, &mp_params); +net_mp_close_rxq (binding->dev, rxq_idx, &mp_params); }percpu_ref_kill (&binding->ref);
@@ -163,7 +163,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx,u32 xa_idx;int err; - err =__net_mp_open_rxq (dev, rxq_idx, &mp_params, extack); + err =net_mp_open_rxq (dev, rxq_idx, &mp_params, extack);if (err)return err;
@@ -176,7 +176,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx,return 0 ; err_close_rxq: -__net_mp_close_rxq (dev, rxq_idx, &mp_params); +net_mp_close_rxq (dev, rxq_idx, &mp_params);return err; }
net/core/netdev_rx_queue.c +5 -24
@@ -191,9 +191,9 @@ int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq_idx) }EXPORT_SYMBOL_NS_GPL (netdev_rx_queue_restart,"NETDEV_INTERNAL" ); -int __net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, -const struct pp_memory_provider_params *p, -struct netlink_ext_ack *extack) +int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack) {const struct netdev_queue_mgmt_ops *qops = dev->queue_mgmt_ops;struct netdev_queue_config qcfg[2 ];
@@ -255,19 +255,8 @@ int __net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx,return ret; } -int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, -struct pp_memory_provider_params *p) -{ -int ret; - -netdev_lock (dev); - ret =__net_mp_open_rxq (dev, rxq_idx, p,NULL ); -netdev_unlock (dev); -return ret; -} - -void __net_mp_close_rxq (struct net_device *dev,unsigned int ifq_idx, -const struct pp_memory_provider_params *old_p) +void net_mp_close_rxq (struct net_device *dev,unsigned int ifq_idx, +const struct pp_memory_provider_params *old_p) {struct netdev_queue_config qcfg[2 ];struct netdev_rx_queue *rxq;
@@ -296,11 +285,3 @@ void __net_mp_close_rxq(struct net_device *dev, unsigned int ifq_idx, err =netdev_rx_queue_reconfig (dev, ifq_idx, &qcfg[0 ], &qcfg[1 ]);WARN_ON (err && err != -ENETDOWN ); } - -void net_mp_close_rxq (struct net_device *dev,unsigned ifq_idx, -struct pp_memory_provider_params *old_p) -{ -netdev_lock (dev); -__net_mp_close_rxq (dev, ifq_idx, old_p); -netdev_unlock (dev); -}
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 05/16] net: Slightly simplify net_mp_{open,close}_rxq
Quoted 4 lines · On Thu, 29 Jan 2026 23:28:19 +0100 Daniel Borkmann wrote:
include/net/page_pool/memory_provider.h +0
-4-int net_mp_open_rxq (struct net_device *dev,unsigned ifq_idx, -struct pp_memory_provider_params *p); -int __net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, - const struct pp_memory_provider_params *p,
include/net/page_pool/memory_provider.h bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr); · line 30
Quoted 10 lines
include/net/page_pool/memory_provider.h +5
-5- struct netlink_ext_ack *extack); -void net_mp_close_rxq (struct net_device *dev,unsigned ifq_idx, -struct pp_memory_provider_params *old_p); -void __net_mp_close_rxq (struct net_device *dev,unsigned int rxq_idx, -const struct pp_memory_provider_params *old_p); +int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack); +void net_mp_close_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *old_p);
SG, but netdevices.rst says:
For the (mostly software) drivers that need to interact with the core stack, there are two sets of interfaces: ``dev_xxx``/``netdev_xxx`` and ``netif_xxx`` (e.g., ``dev_set_mtu`` and ``netif_set_mtu``). The ``dev_xxx``/``netdev_xxx`` functions handle acquiring the instance lock themselves, while the ``netif_xxx`` functions assume that the driver has already acquired the instance lock.
Let's take this opportunity to replace the net_ prefix with netif_, if we want to assume that the caller took the instance lock?
Daniel Borkmann
maintainerPATCH v8 6/16 net: Proxy net_mp_{open,close}_rxq for leased queues
In reply to Daniel BorkmannFrom: David Wei <[email protected]>
When a process in a container wants to setup a memory provider, it will use the virtual netdev and a leased rxq, and call net_mp_{open,close}_rxq to try and restart the queue. At this point, proxy the queue restart on the real rxq in the physical netdev.
For memory providers (io_uring zero-copy rx and devmem), it causes the real rxq in the physical netdev to be filled from a memory provider that has DMA mapped memory from a process within a container.
--- include/net/netdev_rx_queue.h | 4 ++ net/core/netdev_rx_queue.c | 100 +++++++++++++++++++++++++++------- 2 files changed, 83 insertions(+), 21 deletions(-)
include/net/netdev_rx_queue.h +4 -0
@@ -75,4 +75,8 @@ enum netif_lease_dir {struct netdev_rx_queue *__netif_get_rx_queue_lease (struct net_device **dev,unsigned int *rxq,enum netif_lease_dir dir); +struct netdev_rx_queue * +netif_get_rx_queue_lease_locked (struct net_device **dev,unsigned int *rxq); +void netif_put_rx_queue_lease_locked (struct net_device *orig_dev, +struct net_device *dev);#endif /* _LINUX_NETDEV_RX_QUEUE_H */
net/core/netdev_rx_queue.c +79 -21
136 lines not on this page · raw message
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 06/16] net: Proxy net_mp_{open,close}_rxq for leased queues
Quoted 31 lines · On Thu, 29 Jan 2026 23:28:20 +0100 Daniel Borkmann wrote:
net/core/netdev_rx_queue.c +8
-12@@ -191,24 +214,15 @@ int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq_idx) }EXPORT_SYMBOL_NS_GPL (netdev_rx_queue_restart,"NETDEV_INTERNAL" ); -int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, -const struct pp_memory_provider_params *p, -struct netlink_ext_ack *extack) +static int __net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack) {const struct netdev_queue_mgmt_ops *qops = dev->queue_mgmt_ops;struct netdev_queue_config qcfg[2 ];struct netdev_rx_queue *rxq;int ret; -if (!netdev_need_ops_lock (dev)) -return -EOPNOTSUPP ; - -if (rxq_idx >= dev->real_num_rx_queues) { -NL_SET_ERR_MSG (extack,"rx queue index out of range" ); -return -ERANGE ; - } - rxq_idx =array_index_nospec (rxq_idx, dev->real_num_rx_queues); - +int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack) +{ +struct net_device *orig_dev = dev;
net/core/netdev_rx_queue.c int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq_idx)
Quoted 10 lines
net/core/netdev_rx_queue.c +10
-0+int ret; + +if (!netdev_need_ops_lock (dev)) +return -EOPNOTSUPP ; + +if (rxq_idx >= dev->real_num_rx_queues) { +NL_SET_ERR_MSG (extack,"rx queue index out of range" ); +return -ERANGE ; + } +
nit: delete this empty line please, like it was in the original code.
Quoted 6 lines
net/core/netdev_rx_queue.c +6
-0+ rxq_idx =array_index_nospec (rxq_idx, dev->real_num_rx_queues); + +if (!netif_get_rx_queue_lease_locked (&dev, &rxq_idx)) { +NL_SET_ERR_MSG (extack,"rx queue leased to a virtual netdev" ); +return -EBUSY ; + }
net/core/netdev_rx_queue.c int net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, · line 302
Quoted 10 lines
net/core/netdev_rx_queue.c +10
-0+if (!dev->dev.parent) { +NL_SET_ERR_MSG (extack,"rx queue belongs to a virtual netdev" ); + ret = -EOPNOTSUPP ; + goto out; + } + + ret =__net_mp_open_rxq (dev, rxq_idx, p, extack); +out: +netif_put_rx_queue_lease_locked (orig_dev, dev); +return ret;
Please:
if (!netif_queue_is_a_lease()) return __net_mp_open_rxq(dev, rxq_idx, p, extack); ..explicit code that deals with the lease, no conditional locking ... ret = __net_mp_open_rxq($dev, $rxq_idx, p, extack); ...
net/core/netdev_rx_queue.c int net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, · line 312
Quoted 2 lines
net/core/netdev_rx_queue.c +1
-1- rxq =__netif_get_rx_queue (dev, ifq_idx); + rxq =__netif_get_rx_queue (dev, rxq_idx);
Indeed good to cleanup the ifq_idx naming that sneaked in, but IDK if this belongs in this commit :S
Re: [PATCH net-next v8 06/16] net: Proxy net_mp_{open,close}_rxq for leased queues
Quoted 36 lines · Jakub Kicinski
Quoted 35 lines · On Thu, 29 Jan 2026 23:28:20 +0100 Daniel Borkmann wrote:
net/core/netdev_rx_queue.c +12
-12@@ -191,24 +214,15 @@ int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq_idx) }EXPORT_SYMBOL_NS_GPL (netdev_rx_queue_restart,"NETDEV_INTERNAL" ); -int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, -const struct pp_memory_provider_params *p, -struct netlink_ext_ack *extack) +static int __net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack) {const struct netdev_queue_mgmt_ops *qops = dev->queue_mgmt_ops;struct netdev_queue_config qcfg[2 ];struct netdev_rx_queue *rxq;int ret; -if (!netdev_need_ops_lock (dev)) -return -EOPNOTSUPP ; - -if (rxq_idx >= dev->real_num_rx_queues) { -NL_SET_ERR_MSG (extack,"rx queue index out of range" ); -return -ERANGE ; - } - rxq_idx =array_index_nospec (rxq_idx, dev->real_num_rx_queues); - +int net_mp_open_rxq (struct net_device *dev,unsigned int rxq_idx, +const struct pp_memory_provider_params *p, +struct netlink_ext_ack *extack) +{ +struct net_device *orig_dev = dev; +int ret; + +if (!netdev_need_ops_lock (dev)) +return -EOPNOTSUPP ;
Quoted 8 lines
net/core/netdev_rx_queue.c int net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, · line 286
Quoted 6 lines
net/core/netdev_rx_queue.c +6
-0+ +if (rxq_idx >= dev->real_num_rx_queues) { +NL_SET_ERR_MSG (extack,"rx queue index out of range" ); +return -ERANGE ; + } +nit: delete this empty line please, like it was in the original code.
Yes.
Quoted 20 lines
Quoted 7 lines
net/core/netdev_rx_queue.c +6
-0+ rxq_idx =array_index_nospec (rxq_idx, dev->real_num_rx_queues); + +if (!netif_get_rx_queue_lease_locked (&dev, &rxq_idx)) { +NL_SET_ERR_MSG (extack,"rx queue leased to a virtual netdev" ); +return -EBUSY ; + }net/core/netdev_rx_queue.c int net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, · line 302
Quoted 10 lines
net/core/netdev_rx_queue.c +10
-0+if (!dev->dev.parent) { +NL_SET_ERR_MSG (extack,"rx queue belongs to a virtual netdev" ); + ret = -EOPNOTSUPP ; + goto out; + } + + ret =__net_mp_open_rxq (dev, rxq_idx, p, extack); +out: +netif_put_rx_queue_lease_locked (orig_dev, dev); +return ret;Please:
Quoted 7 lines
if (!netif_queue_is_a_lease()) return __net_mp_open_rxq(dev, rxq_idx, p, extack); ..explicit code that deals with the lease, no conditional locking ... ret = __net_mp_open_rxq($dev, $rxq_idx, p, extack); ...
Understood.
Quoted 7 lines
net/core/netdev_rx_queue.c int net_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx, · line 312
Quoted 4 lines
net/core/netdev_rx_queue.c +1
-1- rxq =__netif_get_rx_queue (dev, ifq_idx); + rxq =__netif_get_rx_queue (dev, rxq_idx);Indeed good to cleanup the
ifq_idxnaming that sneaked in, but IDK if this belongs in this commit :S
It really annoyed me haha. I'll remove this from this series and send it as a follow up.
Daniel Borkmann
maintainerPATCH v8 7/16 net: Proxy netdev_queue_get_dma_dev for leased queues
In reply to Daniel BorkmannFrom: David Wei <[email protected]>
Extend netdev_queue_get_dma_dev to return the physical device of the real rxq for DMA in case the queue was leased. This allows memory providers like io_uring zero-copy or devmem to bind to the physically leased rxq via virtual devices such as netkit.
--- net/core/netdev_queues.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
net/core/netdev_queues.c +15 -2
@@ -9,16 +9,29 @@* @dev: net_device * @idx: queue index * -* Get dma device for zero-copy operations to be used for this queue. +* Get dma device for zero-copy operations to be used for this queue. If the +* queue is leased to a physical queue, we retrieve the latter's dma device. * When such device is not available or valid, the function will return NULL. * * Return: Device or NULL on error */ struct device *netdev_queue_get_dma_dev (struct net_device *dev,int idx) { -const struct netdev_queue_mgmt_ops *queue_ops = dev->queue_mgmt_ops; +const struct netdev_queue_mgmt_ops *queue_ops;struct device *dma_dev; +if (idx < dev->real_num_rx_queues) { +struct netdev_rx_queue *rxq =__netif_get_rx_queue (dev, idx); + +if (rxq->lease) { + rxq = rxq->lease; + dev = rxq->dev; + idx =get_netdev_rx_queue_index (rxq); + } + } + + queue_ops = dev->queue_mgmt_ops; +if (queue_ops && queue_ops->ndo_queue_get_dma_dev) dma_dev = queue_ops->ndo_queue_get_dma_dev (dev, idx);else
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 07/16] net: Proxy netdev_queue_get_dma_dev for leased queues
Quoted 14 lines · On Thu, 29 Jan 2026 23:28:21 +0100 Daniel Borkmann wrote:
- * Get dma device for zero-copy operations to be used for this queue.
+ * Get dma device for zero-copy operations to be used for this queue. If the
+ * queue is leased to a physical queue, we retrieve the latter's dma device.
* When such device is not available or valid, the function will return NULL. *
* Return: Device or NULL on error
*/
struct device *netdev_queue_get_dma_dev(structnet_device*dev, int idx) {net/core/netdev_queues.c +2
-1-const struct netdev_queue_mgmt_ops *queue_ops = dev->queue_mgmt_ops; +const struct netdev_queue_mgmt_ops *queue_ops;struct device *dma_dev; +if (idx < dev->real_num_rx_queues) {
net/core/netdev_queues.c line 33
Quoted 10 lines
net/core/netdev_queues.c +10
-0+struct netdev_rx_queue *rxq =__netif_get_rx_queue (dev, idx); + +if (rxq->lease) { + rxq = rxq->lease; + dev = rxq->dev; + idx =get_netdev_rx_queue_index (rxq); + } + } + + queue_ops = dev->queue_mgmt_ops;
We're gonna call physical device's op without locking it no?
Re: [PATCH net-next v8 07/16] net: Proxy netdev_queue_get_dma_dev for leased queues
Quoted 14 lines · Jakub Kicinski
Quoted 13 lines · On Thu, 29 Jan 2026 23:28:21 +0100 Daniel Borkmann wrote:
- * Get dma device for zero-copy operations to be used for this queue.
+ * Get dma device for zero-copy operations to be used for this queue. If the
+ * queue is leased to a physical queue, we retrieve the latter's dma device.
* When such device is not available or valid, the function will return NULL. *
* Return: Device or NULL on error
*/
struct device *netdev_queue_get_dma_dev(structnet_device*dev, int idx) {net/core/netdev_queues.c +1
-1-const struct netdev_queue_mgmt_ops *queue_ops = dev->queue_mgmt_ops; +const struct netdev_queue_mgmt_ops *queue_ops;struct device *dma_dev;
Quoted 13 lines
net/core/netdev_queues.c line 33
Quoted 11 lines
net/core/netdev_queues.c +11
-0+if (idx < dev->real_num_rx_queues) { +struct netdev_rx_queue *rxq =__netif_get_rx_queue (dev, idx); + +if (rxq->lease) { + rxq = rxq->lease; + dev = rxq->dev; + idx =get_netdev_rx_queue_index (rxq); + } + } + + queue_ops = dev->queue_mgmt_ops;We're gonna call physical device's op without locking it no?
Yeah, sorry about that. I'll change this to be similar to the new mp_open.
Daniel Borkmann
maintainerPATCH v8 8/16 xsk: Extend xsk_rcv_check validation
In reply to Daniel Borkmannxsk_rcv_check tests for inbound packets to see whether they match the bound AF_XDP socket. Refactor the test into a small helper xsk_dev_queue_valid and move the validation against xs->dev and xs->queue_id there.
The fast-path case stays in place and allows for quick return in xsk_dev_queue_valid. If it fails, the validation is extended to check whether the AF_XDP socket is bound against a leased queue, and if the case then the test is redone.
--- net/xdp/xsk.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-)
net/xdp/xsk.c +26 -3
@@ -324,14 +324,37 @@ static bool xsk_is_bound(struct xdp_sock *xs)return false; } +static bool xsk_dev_queue_valid (const struct xdp_sock *xs, +const struct xdp_rxq_info *info) +{ +struct net_device *dev = xs->dev; +u32 queue_index = xs->queue_id; +struct netdev_rx_queue *rxq; + +if (info->dev == dev && + info->queue_index == queue_index) +return true; + +if (queue_index < dev->real_num_rx_queues) { + rxq =READ_ONCE (__netif_get_rx_queue (dev, queue_index)->lease); +if (!rxq) +return false; + + dev = rxq->dev; + queue_index =get_netdev_rx_queue_index (rxq); + +return info->dev == dev && + info->queue_index == queue_index; + } +return false; +} +static int xsk_rcv_check (struct xdp_sock *xs,struct xdp_buff *xdp,u32 len) {if (!xsk_is_bound (xs))return -ENXIO ; - -if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index) +if (!xsk_dev_queue_valid (xs, xdp->rxq))return -EINVAL ; -if (len >xsk_pool_get_rx_frame_size (xs->pool) && !xs->sg) { xs->rx_dropped++;return -ENOSPC ;
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 9/16 xsk: Proxy pool management for leased queues
In reply to Daniel BorkmannSimilarly to the net_mp_{open,close}_rxq handling for leased queues, proxy the xsk_{reg,clear}_pool_at_qid via netif_get_rx_queue_lease_locked such that in case a virtual netdev picked a leased rxq, the request gets through to the real rxq in the physical netdev. The proxying is only relevant for queue_id < dev->real_num_rx_queues since right now its only supported for rxqs.
--- net/xdp/xsk.c | 48 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 13 deletions(-)
net/xdp/xsk.c +35 -13
@@ -23,6 +23,8 @@#include <linux/netdevice.h> #include <linux/rculist.h> #include <linux/vmalloc.h> + +#include <net/netdev_queues.h> #include <net/xdp_sock_drv.h> #include <net/busy_poll.h> #include <net/netdev_lock.h>
@@ -117,10 +119,18 @@ EXPORT_SYMBOL(xsk_get_pool_from_qid);void xsk_clear_pool_at_qid (struct net_device *dev,u16 queue_id) { -if (queue_id < dev->num_rx_queues) - dev->_rx[queue_id].pool =NULL ; -if (queue_id < dev->num_tx_queues) - dev->_tx[queue_id].pool =NULL ; +struct net_device *orig_dev = dev; +unsigned int id = queue_id; + +if (id < dev->real_num_rx_queues) +WARN_ON_ONCE (!netif_get_rx_queue_lease_locked (&dev, &id)); + +if (id < dev->real_num_rx_queues) + dev->_rx[id].pool =NULL ; +if (id < dev->real_num_tx_queues) + dev->_tx[id].pool =NULL ; + +netif_put_rx_queue_lease_locked (orig_dev, dev); }/* The buffer pool is stored both in the _rx struct and the _tx struct as we do
@@ -130,17 +140,29 @@ void xsk_clear_pool_at_qid(struct net_device *dev, u16 queue_id)int xsk_reg_pool_at_qid (struct net_device *dev,struct xsk_buff_pool *pool,u16 queue_id) { - if (queue_id >= max_t(unsigned int, - dev->real_num_rx_queues, - dev->real_num_tx_queues)) -return -EINVAL ; +struct net_device *orig_dev = dev; +unsigned int id = queue_id; +int ret =0 ; -if (queue_id < dev->real_num_rx_queues) - dev->_rx[queue_id].pool = pool; -if (queue_id < dev->real_num_tx_queues) - dev->_tx[queue_id].pool = pool; +if (id >=max (dev->real_num_rx_queues, + dev->real_num_tx_queues)) +return -EINVAL ; +if (id < dev->real_num_rx_queues) { +if (!netif_get_rx_queue_lease_locked (&dev, &id)) +return -EBUSY ; +if (xsk_get_pool_from_qid (dev, id)) { + ret = -EBUSY ; + goto out; + } + } -return 0 ; +if (id < dev->real_num_rx_queues) + dev->_rx[id].pool = pool; +if (id < dev->real_num_tx_queues) + dev->_tx[id].pool = pool; +out: +netif_put_rx_queue_lease_locked (orig_dev, dev); +return ret; } static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff_xsk *xskb, u32 len,
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 10/16 netkit: Add single device mode for netkit
In reply to Daniel BorkmannAdd a single device mode for netkit instead of netkit pairs. The primary target for the paired devices is to connect network namespaces, of course, and support has been implemented in projects like Cilium [0]. For the rxq leasing the plan is to support two main scenarios related to single device mode:
* For the use-case of io_uring zero-copy, the control plane can either set up a netkit pair where the peer device can perform rxq leasing which is then tied to the lifetime of the peer device, or the control plane can use a regular netkit pair to connect the hostns to a Pod/container and dynamically add/remove rxq leasing through a single device without having to interrupt the device pair. In the case of io_uring, the memory pool is used as skb non-linear pages, and thus the skb will go its way through the regular stack into netkit. Things like the netkit policy when no BPF is attached or skb scrubbing etc apply as-is in case the paired devices are used, or if the backend memory is tied to the single device and traffic goes through a paired device.
* For the use-case of AF_XDP, the control plane needs to use netkit in the single device mode. The single device mode currently enforces only a pass policy when no BPF is attached, and does not yet support BPF link attachments for AF_XDP. skbs sent to that device get dropped at the moment. Given AF_XDP operates at a lower layer of the stack tying this to the netkit pair did not make sense. In future, the plan is to allow BPF at the XDP layer which can: i) process traffic coming from the AF_XDP application (e.g. QEMU with AF_XDP backend) to filter egress traffic or to push selected egress traffic up to the single netkit device to the local stack (e.g. DHCP requests), and ii) vice-versa skbs sent to the single netkit into the AF_XDP application (e.g. DHCP replies). Also, the control-plane can dynamically manage rxq leasing for the single netkit device without having to interrupt (e.g. down/up cycle) the main netkit pair for the Pod which has traffic going in and out.
--- drivers/net/netkit.c | 110 ++++++++++++++++++++++------------- include/uapi/linux/if_link.h | 6 ++ 2 files changed, 76 insertions(+), 40 deletions(-)
drivers/net/netkit.c +70 -40
@@ -26,6 +26,7 @@ struct netkit {__cacheline_group_begin (netkit_slowpath);enum netkit_mode mode; +enum netkit_pairing pair;bool primary;u32 headroom;__cacheline_group_end (netkit_slowpath);
@@ -135,6 +136,10 @@ static int netkit_open(struct net_device *dev)struct netkit *nk =netkit_priv (dev);struct net_device *peer =rtnl_dereference (nk->peer); +if (nk->pair ==NETKIT_DEVICE_SINGLE ) { +netif_carrier_on (dev); +return 0 ; + }if (!peer)return -ENOTCONN ;if (peer->flags &IFF_UP ) {
@@ -335,6 +340,7 @@ static int netkit_new_link(struct net_device *dev,enum netkit_scrub scrub_prim =NETKIT_SCRUB_DEFAULT ;enum netkit_scrub scrub_peer =NETKIT_SCRUB_DEFAULT ;struct nlattr *peer_tb[IFLA_MAX +1 ], **tbp, *attr; +enum netkit_pairing pair =NETKIT_DEVICE_PAIR ;enum netkit_action policy_prim =NETKIT_PASS ;enum netkit_action policy_peer =NETKIT_PASS ;struct nlattr **data = params->data;
@@ -343,7 +349,8 @@ static int netkit_new_link(struct net_device *dev,struct nlattr **tb = params->tb;u16 headroom =0 , tailroom =0 ;struct ifinfomsg *ifmp =NULL ; -struct net_device *peer; +struct net_device *peer =NULL ; +bool seen_peer = false;char ifname[IFNAMSIZ ];struct netkit *nk;int err;
@@ -380,6 +387,12 @@ static int netkit_new_link(struct net_device *dev, headroom =nla_get_u16 (data[IFLA_NETKIT_HEADROOM ]);if (data[IFLA_NETKIT_TAILROOM ]) tailroom =nla_get_u16 (data[IFLA_NETKIT_TAILROOM ]); +if (data[IFLA_NETKIT_PAIRING ]) + pair =nla_get_u32 (data[IFLA_NETKIT_PAIRING ]); + + seen_peer = data[IFLA_NETKIT_PEER_INFO ] || + data[IFLA_NETKIT_PEER_SCRUB ] || + data[IFLA_NETKIT_PEER_POLICY ]; }if (ifmp && tbp[IFLA_IFNAME ]) {
@@ -392,45 +405,46 @@ static int netkit_new_link(struct net_device *dev,if (mode !=NETKIT_L2 && (tb[IFLA_ADDRESS ] || tbp[IFLA_ADDRESS ]))return -EOPNOTSUPP ; +if (pair ==NETKIT_DEVICE_SINGLE && + (tb != tbp || seen_peer || policy_prim !=NETKIT_PASS )) +return -EOPNOTSUPP ; - peer =rtnl_create_link (peer_net, ifname, ifname_assign_type, - &netkit_link_ops, tbp, extack); -if (IS_ERR (peer)) -return PTR_ERR (peer); - -netif_inherit_tso_max (peer, dev); -if (headroom) { - peer->needed_headroom = headroom; - dev->needed_headroom = headroom; - } -if (tailroom) { - peer->needed_tailroom = tailroom; - dev->needed_tailroom = tailroom; - } - -if (mode ==NETKIT_L2 && !(ifmp && tbp[IFLA_ADDRESS ])) -eth_hw_addr_random (peer); -if (ifmp && dev->ifindex) - peer->ifindex = ifmp->ifi_index; - - nk =netkit_priv (peer); - nk->primary = false; - nk->policy = policy_peer; - nk->scrub = scrub_peer; - nk->mode = mode; - nk->headroom = headroom; -bpf_mprog_bundle_init (&nk->bundle); +if (pair ==NETKIT_DEVICE_PAIR ) { + peer =rtnl_create_link (peer_net, ifname, ifname_assign_type, + &netkit_link_ops, tbp, extack); +if (IS_ERR (peer)) +return PTR_ERR (peer); + +netif_inherit_tso_max (peer, dev); +if (headroom) + peer->needed_headroom = headroom; +if (tailroom) + peer->needed_tailroom = tailroom; +if (mode ==NETKIT_L2 && !(ifmp && tbp[IFLA_ADDRESS ])) +eth_hw_addr_random (peer); +if (ifmp && dev->ifindex) + peer->ifindex = ifmp->ifi_index; - err =register_netdevice (peer); -if (err <0 ) - goto err_register_peer; -netif_carrier_off (peer); -if (mode ==NETKIT_L2 ) -dev_change_flags (peer, peer->flags & ~IFF_NOARP ,NULL ); + nk =netkit_priv (peer); + nk->primary = false; + nk->policy = policy_peer; + nk->scrub = scrub_peer; + nk->mode = mode; + nk->pair = pair; + nk->headroom = headroom; +bpf_mprog_bundle_init (&nk->bundle); + + err =register_netdevice (peer); +if (err <0 ) + goto err_register_peer; +netif_carrier_off (peer); +if (mode ==NETKIT_L2 ) +dev_change_flags (peer, peer->flags & ~IFF_NOARP ,NULL ); - err =rtnl_configure_link (peer,NULL ,0 ,NULL ); -if (err <0 ) - goto err_configure_peer; + err =rtnl_configure_link (peer,NULL ,0 ,NULL ); +if (err <0 ) + goto err_configure_peer; + }if (mode ==NETKIT_L2 && !tb[IFLA_ADDRESS ])eth_hw_addr_random (dev);
@@ -438,12 +452,17 @@ static int netkit_new_link(struct net_device *dev,nla_strscpy (dev->name, tb[IFLA_IFNAME ],IFNAMSIZ ); else strscpy(dev->name, "nk%d", IFNAMSIZ); +if (headroom) + dev->needed_headroom = headroom; +if (tailroom) + dev->needed_tailroom = tailroom; nk =netkit_priv (dev); nk->primary = true; nk->policy = policy_prim; nk->scrub = scrub_prim; nk->mode = mode; + nk->pair = pair; nk->headroom = headroom;bpf_mprog_bundle_init (&nk->bundle);
@@ -455,10 +474,12 @@ static int netkit_new_link(struct net_device *dev,dev_change_flags (dev, dev->flags & ~IFF_NOARP ,NULL );rcu_assign_pointer (netkit_priv (dev)->peer, peer); -rcu_assign_pointer (netkit_priv (peer)->peer, dev); +if (peer) +rcu_assign_pointer (netkit_priv (peer)->peer, dev);return 0 ; err_configure_peer: -unregister_netdevice (peer); +if (peer) +unregister_netdevice (peer);return err; err_register_peer:free_netdev (peer);
@@ -518,6 +539,8 @@ static struct net_device *netkit_dev_fetch(struct net *net, u32 ifindex, u32 whi nk =netkit_priv (dev);if (!nk->primary)return ERR_PTR (-EACCES ); +if (nk->pair ==NETKIT_DEVICE_SINGLE ) +return ERR_PTR (-EOPNOTSUPP );if (which ==BPF_NETKIT_PEER ) { dev =rcu_dereference_rtnl (nk->peer);if (!dev)
@@ -879,6 +902,7 @@ static int netkit_change_link(struct net_device *dev, struct nlattr *tb[], { IFLA_NETKIT_PEER_INFO, "peer info" }, { IFLA_NETKIT_HEADROOM, "headroom" }, { IFLA_NETKIT_TAILROOM, "tailroom" }, + { IFLA_NETKIT_PAIRING, "pairing" }, };if (!nk->primary) {
@@ -898,9 +922,11 @@ static int netkit_change_link(struct net_device *dev, struct nlattr *tb[], }if (data[IFLA_NETKIT_POLICY ]) { + err = -EOPNOTSUPP ; attr = data[IFLA_NETKIT_POLICY ]; policy =nla_get_u32 (attr); - err =netkit_check_policy (policy, attr, extack); +if (nk->pair ==NETKIT_DEVICE_PAIR ) + err =netkit_check_policy (policy, attr, extack);if (err)return err;WRITE_ONCE (nk->policy, policy);
@@ -931,6 +957,7 @@ static size_t netkit_get_size(const struct net_device *dev)nla_total_size (sizeof (u8)) +/* IFLA_NETKIT_PRIMARY */ nla_total_size (sizeof (u16)) +/* IFLA_NETKIT_HEADROOM */ nla_total_size (sizeof (u16)) +/* IFLA_NETKIT_TAILROOM */ +nla_total_size (sizeof (u32)) +/* IFLA_NETKIT_PAIRING */ 0 ; }
@@ -951,6 +978,8 @@ static int netkit_fill_info(struct sk_buff *skb, const struct net_device *dev)return -EMSGSIZE ;if (nla_put_u16 (skb,IFLA_NETKIT_TAILROOM , dev->needed_tailroom))return -EMSGSIZE ; +if (nla_put_u32 (skb,IFLA_NETKIT_PAIRING , nk->pair)) +return -EMSGSIZE ;if (peer) { nk =netkit_priv (peer);
@@ -972,6 +1001,7 @@ static const struct nla_policy netkit_policy[IFLA_NETKIT_MAX + 1] = { [IFLA_NETKIT_TAILROOM] = { .type = NLA_U16 }, [IFLA_NETKIT_SCRUB] = NLA_POLICY_MAX(NLA_U32, NETKIT_SCRUB_DEFAULT), [IFLA_NETKIT_PEER_SCRUB] = NLA_POLICY_MAX(NLA_U32, NETKIT_SCRUB_DEFAULT), + [IFLA_NETKIT_PAIRING] = NLA_POLICY_MAX(NLA_U32, NETKIT_DEVICE_SINGLE), [IFLA_NETKIT_PRIMARY ] = { .type =NLA_REJECT , .reject_message = "Primary attribute is read-only" }, };
include/uapi/linux/if_link.h +6 -0
@@ -1296,6 +1296,11 @@ enum netkit_mode { NETKIT_L3, }; +enum netkit_pairing { +NETKIT_DEVICE_PAIR , +NETKIT_DEVICE_SINGLE , +}; +/* NETKIT_SCRUB_NONE leaves clearing skb->{mark,priority} up to * the BPF program if attached. This also means the latter can * consume the two fields if they were populated earlier.
@@ -1320,6 +1325,7 @@ enum {IFLA_NETKIT_PEER_SCRUB ,IFLA_NETKIT_HEADROOM ,IFLA_NETKIT_TAILROOM , +IFLA_NETKIT_PAIRING , __IFLA_NETKIT_MAX, };#define IFLA_NETKIT_MAX (__IFLA_NETKIT_MAX - 1)
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 11/16 netkit: Implement rtnl_link_ops->alloc and ndo_queue_create
In reply to Daniel BorkmannFrom: David Wei <[email protected]>
Implement rtnl_link_ops->alloc that allows the number of rx queues to be set when netkit is created. By default, netkit has only a single rxq (and single txq). The number of queues is deliberately not allowed to be changed via ethtool -L and is fixed for the lifetime of a netkit instance.
For netkit device creation, numrxqueues with larger than one rxq can be specified. These rxqs are leasable to real rxqs in physical netdevs:
ip link add type netkit peer numrxqueues 64 # for device pair
ip link add numrxqueues 64 type netkit single # for single device
The limit of numrxqueues for netkit is currently set to 1024, which allows leasing multiple real rxqs from physical netdevs.
The implementation of ndo_queue_create() adds a new rxq during the queue lease operation. We allow to create queues either in single device mode or for the case of dual device mode for the netkit peer device which gets placed into the target network namespace. For dual device mode the lease against the primary device does not make sense for the targeted use cases, and therefore gets rejected.
We also need to add a lockdep class for netkit, such that lockdep does not trip over us, similarly done as in commit 0bef512012b1 ("net: add netdev_lockdep_set_classes() to virtual drivers").
This is also the last missing bit to netkit for supporting io_uring with zero-copy mode [0]. Up until this point it was not possible to consume the latter out of containers or Kubernetes Pods where applications are in their own network namespace.
io_uring example with eth0 being a physical device with 16 queues where netkit is bound to the last queue, iou-zcrx.c is binary from selftests. Flow steering to that queue is based on the service VIP:port of the server utilizing io_uring:
# ethtool -X eth0 start 0 equal 15
# ethtool -X eth0 start 15 equal 1 context new
# ethtool --config-ntuple eth0 flow-type tcp4 dst-ip 1.2.3.4 dst-port 5000 action 15
# ip netns add foo
# ip link add type netkit peer numrxqueues 2
# ./pyynl/cli.py --spec ~/netlink/specs/netdev.yaml \
--do queue-create \
--json "{"ifindex": $(ifindex nk0), "type": "rx", \
"lease": { "ifindex": $(ifindex eth0), \
"queue": { "type": "rx", "id": 15 } } }"
{'id': 1}
# ip link set nk0 netns foo
# ip link set nk1 up
# ip netns exec foo ip link set lo up
# ip netns exec foo ip link set nk0 up
# ip netns exec foo ip addr add 1.2.3.4/32 dev nk0
[ ... setup routing etc to get external traffic into the netns ... ]
# ip netns exec foo ./iou-zcrx -s -p 5000 -i nk0 -q 1Remote io_uring client:
# ./iou-zcrx -c -h 1.2.3.4 -p 5000 -l 12840 -z 65536
We have tested the above against a Broadcom BCM957504 (bnxt_en) 100G NIC, supporting TCP header/data split.
Similarly, this also works for devmem which we tested using ncdevmem:
# ip netns exec foo ./ncdevmem -s 1.2.3.4 -l -p 5000 -f nk0 -t 1 -q 1
And on the remote client:
# ./ncdevmem -s 1.2.3.4 -p 5000 -f eth0
For Cilium, the plan is to open up support for the various memory providers for regular Kubernetes Pods when Cilium is configured with netkit datapath mode.
--- drivers/net/netkit.c | 117 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 105 insertions(+), 12 deletions(-)
drivers/net/netkit.c +105 -12
@@ -9,11 +9,20 @@#include <linux/bpf_mprog.h> #include <linux/indirect_call_wrapper.h> +#include <net/netdev_lock.h> +#include <net/netdev_queues.h> +#include <net/netdev_rx_queue.h> #include <net/netkit.h> #include <net/dst.h> #include <net/tcx.h> -#define DRV_NAME "netkit" +#define NETKIT_DRV_NAME "netkit" + +#define NETKIT_NUM_RX_QUEUES_MAX 1024 +#define NETKIT_NUM_TX_QUEUES_MAX 1 + +#define NETKIT_NUM_RX_QUEUES_REAL 1 +#define NETKIT_NUM_TX_QUEUES_REAL 1struct netkit {__cacheline_group_begin (netkit_fastpath);
@@ -37,6 +46,8 @@ struct netkit_link {struct net_device *dev; }; +static struct rtnl_link_ops netkit_link_ops; + static __always_inline intnetkit_run (const struct bpf_mprog_entry *entry,struct sk_buff *skb,enum netkit_action ret)
@@ -224,9 +235,16 @@ static void netkit_get_stats(struct net_device *dev, stats->tx_dropped =DEV_STATS_READ (dev, tx_dropped); } +static int netkit_init (struct net_device *dev) +{ +netdev_lockdep_set_classes (dev); +return 0 ; +} +static void netkit_uninit (struct net_device *dev);static const struct net_device_ops netkit_netdev_ops = { + .ndo_init = netkit_init, .ndo_open = netkit_open, .ndo_stop = netkit_close, .ndo_start_xmit = netkit_xmit,
@@ -243,13 +261,87 @@ static const struct net_device_ops netkit_netdev_ops = {static void netkit_get_drvinfo (struct net_device *dev,struct ethtool_drvinfo *info) { -strscpy (info->driver,DRV_NAME ,sizeof (info->driver)); +strscpy (info->driver,NETKIT_DRV_NAME ,sizeof (info->driver)); }static const struct ethtool_ops netkit_ethtool_ops = { .get_drvinfo = netkit_get_drvinfo, }; +static int netkit_queue_create (struct net_device *dev) +{ +struct netkit *nk =netkit_priv (dev); +u32 rxq_count_old, rxq_count_new; +int err; + + rxq_count_old = dev->real_num_rx_queues; + rxq_count_new = rxq_count_old +1 ; + +/* Only allow to lease a queue in single device mode or to +* lease against the peer device which then ends up in the +* target netns. +*/ +if (nk->pair ==NETKIT_DEVICE_PAIR && nk->primary) +return -EOPNOTSUPP ; + +if (netif_running (dev)) +netif_carrier_off (dev); + err =netif_set_real_num_rx_queues (dev, rxq_count_new); +if (netif_running (dev)) +netif_carrier_on (dev); + +return err ? : rxq_count_old; +} + +static const struct netdev_queue_mgmt_ops netkit_queue_mgmt_ops = { + .ndo_queue_create = netkit_queue_create, +}; + +static struct net_device *netkit_alloc (struct nlattr *tb[], +const char *ifname, +unsigned char name_assign_type, +unsigned int num_tx_queues, +unsigned int num_rx_queues) +{ +const struct rtnl_link_ops *ops = &netkit_link_ops; +struct net_device *dev; + +if (num_tx_queues >NETKIT_NUM_TX_QUEUES_MAX || + num_rx_queues >NETKIT_NUM_RX_QUEUES_MAX ) +return ERR_PTR (-EOPNOTSUPP ); + + dev =alloc_netdev_mqs (ops->priv_size, ifname, + name_assign_type, ops->setup, + num_tx_queues, num_rx_queues); +if (dev) { + dev->real_num_tx_queues =NETKIT_NUM_TX_QUEUES_REAL ; + dev->real_num_rx_queues =NETKIT_NUM_RX_QUEUES_REAL ; + } +return dev; +} + +static void netkit_queue_unlease (struct net_device *dev) +{ +struct netdev_rx_queue *rxq, *rxq_lease; +struct net_device *dev_lease; +int i; + +if (dev->real_num_rx_queues ==1 ) +return ; + +netdev_lock (dev); +for (i =1 ; i < dev->real_num_rx_queues; i++) { + rxq =__netif_get_rx_queue (dev, i); + rxq_lease = rxq->lease; + dev_lease = rxq_lease->dev; + +netdev_lock (dev_lease); +netdev_rx_queue_unlease (rxq, rxq_lease); +netdev_unlock (dev_lease); + } +netdev_unlock (dev); +} +static void netkit_setup (struct net_device *dev) { static const netdev_features_t netkit_features_hw_vlan =
@@ -280,8 +372,9 @@ static void netkit_setup(struct net_device *dev) dev->priv_flags |=IFF_DISABLE_NETPOLL ; dev->lltx = true; - dev->ethtool_ops = &netkit_ethtool_ops; - dev->netdev_ops = &netkit_netdev_ops; + dev->netdev_ops = &netkit_netdev_ops; + dev->ethtool_ops = &netkit_ethtool_ops; + dev->queue_mgmt_ops = &netkit_queue_mgmt_ops; dev->features |= netkit_features; dev->hw_features = netkit_features;
@@ -330,8 +423,6 @@ static int netkit_validate(struct nlattr *tb[], struct nlattr *data[],return 0 ; } -static struct rtnl_link_ops netkit_link_ops; -static int netkit_new_link (struct net_device *dev,struct rtnl_newlink_params *params,struct netlink_ext_ack *extack)
@@ -867,6 +958,7 @@ static void netkit_release_all(struct net_device *dev)static void netkit_uninit (struct net_device *dev) {netkit_release_all (dev); +netkit_queue_unlease (dev); }static void netkit_del_link (struct net_device *dev,struct list_head *head)
@@ -1007,8 +1099,9 @@ static const struct nla_policy netkit_policy[IFLA_NETKIT_MAX + 1] = { };static struct rtnl_link_ops netkit_link_ops = { - .kind =DRV_NAME , + .kind =NETKIT_DRV_NAME , .priv_size =sizeof (struct netkit ), + .alloc = netkit_alloc, .setup = netkit_setup, .newlink = netkit_new_link, .dellink = netkit_del_link,
@@ -1022,7 +1115,7 @@ static struct rtnl_link_ops netkit_link_ops = { .maxtype = IFLA_NETKIT_MAX, }; -static __init int netkit_init (void ) +static __init int netkit_mod_init (void ) { BUILD_BUG_ON((int)NETKIT_NEXT != (int)TCX_NEXT || (int)NETKIT_PASS != (int)TCX_PASS ||
@@ -1032,16 +1125,16 @@ static __init int netkit_init(void)return rtnl_link_register (&netkit_link_ops); } -static __exit void netkit_exit(void) +static __exit void netkit_mod_exit(void) {rtnl_link_unregister (&netkit_link_ops); } -module_init (netkit_init); -module_exit (netkit_exit); +module_init (netkit_mod_init); +module_exit (netkit_mod_exit);MODULE_DESCRIPTION ("BPF-programmable network device" );MODULE_AUTHOR ("Daniel Borkmann <[email protected]>" );MODULE_AUTHOR ("Nikolay Aleksandrov <[email protected]>" );MODULE_LICENSE ("GPL" ); -MODULE_ALIAS_RTNL_LINK (DRV_NAME ); +MODULE_ALIAS_RTNL_LINK (NETKIT_DRV_NAME );
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 11/16] netkit: Implement rtnl_link_ops->alloc and ndo_queue_create
drivers/net/netkit.c static const struct net_device_ops netkit_netdev_ops = { · line 291
Quoted 5 lines · On Thu, 29 Jan 2026 23:28:25 +0100 Daniel Borkmann wrote:
drivers/net/netkit.c +5
-0+if (netif_running (dev)) +netif_carrier_off (dev); + err =netif_set_real_num_rx_queues (dev, rxq_count_new); +if (netif_running (dev)) +netif_carrier_on (dev);
Why do we need to flap the carrier?
(If there's a reason perhaps add a comment?)
Re: [PATCH net-next v8 11/16] netkit: Implement rtnl_link_ops->alloc and ndo_queue_create
Quoted 9 lines · Jakub Kicinski
drivers/net/netkit.c static const struct net_device_ops netkit_netdev_ops = { · line 291
Quoted 5 lines · On Thu, 29 Jan 2026 23:28:25 +0100 Daniel Borkmann wrote:
drivers/net/netkit.c +5
-0+if (netif_running (dev)) +netif_carrier_off (dev); + err =netif_set_real_num_rx_queues (dev, rxq_count_new); +if (netif_running (dev)) +netif_carrier_on (dev);Why do we need to flap the carrier?
(If there's a reason perhaps add a comment?)
I saw veth doing this in veth_set_channels(): off, set num rx queues, then on. But it doesn't sound like there's actually a need? I'll test without this and remove it if everything works.
Daniel Borkmann
maintainerPATCH v8 12/16 netkit: Add netkit notifier to check for unregistering devices
In reply to Daniel BorkmannAdd a netdevice notifier in netkit to watch for NETDEV_UNREGISTER events. If the target device is indeed NETREG_UNREGISTERING and previously leased a queue to a netkit device, then collect the related netkit devices and batch-unregister_netdevice_many() them.
If this would not be done, then the netkit device would hold a reference on the physical device preventing it from going away. However, in case of both io_uring zero-copy as well as AF_XDP this situation is handled gracefully and the allocated resources are torn down.
In the case where mentioned infra is used through netkit, the applications have a reference on netkit, and netkit in turn holds a reference on the physical device. In order to have netkit release the reference on the physical device, we need such watcher to then unregister the netkit ones.
This is generally quite similar to the dependency handling in case of tunnels (e.g. vxlan bound to a underlying netdev) where the tunnel device gets removed along with the physical device.
# ip a
[...]
4: enp10s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e8:eb:d3:a3:43:f6 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 scope global enp10s0f0np0
valid_lft forever preferred_lft forever
[...]
8: nk@NONE: <BROADCAST,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
[...]# rmmod mlx5_ib # rmmod mlx5_core
[ 309.261822] mlx5_core 0000:0a:00.0 mlx5_0: Port: 1 Link DOWN [ 344.235236] mlx5_core 0000:0a:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0) [ 344.246948] mlx5_core 0000:0a:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0) [ 344.463754] mlx5_core 0000:0a:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0) [ 344.770155] mlx5_core 0000:0a:00.1: E-Switch: cleanup [ 345.345709] mlx5_core 0000:0a:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0) [ 345.357524] mlx5_core 0000:0a:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0) [ 350.995989] mlx5_core 0000:0a:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0) [ 351.574396] mlx5_core 0000:0a:00.0: E-Switch: cleanup
# ip a [...] [ both enp10s0f0np0 and nk gone ] [...]
--- drivers/net/netkit.c | 57 ++++++++++++++++++++++++++++++++++++++- include/linux/netdevice.h | 6 +++++ 2 files changed, 62 insertions(+), 1 deletion(-)
drivers/net/netkit.c +56 -1
@@ -1039,6 +1039,48 @@ static int netkit_change_link(struct net_device *dev, struct nlattr *tb[],return 0 ; } +static void netkit_check_lease_unregister (struct net_device *dev) +{ +LIST_HEAD (list_kill); +u32 q_idx; + +if (READ_ONCE (dev->reg_state) !=NETREG_UNREGISTERING || + !dev->dev.parent) +return ; + +netdev_lock_ops (dev); +for (q_idx =0 ; q_idx < dev->real_num_rx_queues; q_idx++) { +struct net_device *tmp = dev; +u32 tmp_q_idx = q_idx; + +if (netif_rx_queue_lease_get_owner (&tmp, &tmp_q_idx)) { +if (tmp->netdev_ops != &netkit_netdev_ops) +continue ; +/* A single phys device can have multiple queues leased +* to one netkit device. We can only queue that netkit +* device once to the list_kill. Queues of that phys +* device can be leased with different individual netkit +* devices, hence we batch via list_kill. +*/ +if (unregister_netdevice_queued (tmp)) +continue ; +netkit_del_link (tmp, &list_kill); + } + } +netdev_unlock_ops (dev); +unregister_netdevice_many (&list_kill); +} + +static int netkit_notifier (struct notifier_block *this, +unsigned long event,void *ptr) +{ +struct net_device *dev =netdev_notifier_info_to_dev (ptr); + +if (event ==NETDEV_UNREGISTER ) +netkit_check_lease_unregister (dev); +return NOTIFY_DONE ; +} +static size_t netkit_get_size (const struct net_device *dev) { return nla_total_size(sizeof(u32)) + /* IFLA_NETKIT_POLICY */
@@ -1115,18 +1157,31 @@ static struct rtnl_link_ops netkit_link_ops = { .maxtype = IFLA_NETKIT_MAX, }; +static struct notifier_block netkit_netdev_notifier = { + .notifier_call = netkit_notifier, +}; +static __init int netkit_mod_init (void ) { +int ret; +BUILD_BUG_ON ((int )NETKIT_NEXT != (int )TCX_NEXT || (int )NETKIT_PASS != (int )TCX_PASS || (int )NETKIT_DROP != (int )TCX_DROP || (int )NETKIT_REDIRECT != (int )TCX_REDIRECT ); -return rtnl_link_register (&netkit_link_ops); + ret =rtnl_link_register (&netkit_link_ops); +if (ret) +return ret; + ret =register_netdevice_notifier (&netkit_netdev_notifier); +if (ret) +rtnl_link_unregister (&netkit_link_ops); +return ret; } static __exit void netkit_mod_exit(void) { +unregister_netdevice_notifier (&netkit_netdev_notifier);rtnl_link_unregister (&netkit_link_ops); }
include/linux/netdevice.h +6 -0
@@ -3403,11 +3403,17 @@ static inline int dev_direct_xmit(struct sk_buff *skb, u16 queue_id)int register_netdevice (struct net_device *dev);void unregister_netdevice_queue (struct net_device *dev,struct list_head *head);void unregister_netdevice_many (struct list_head *head); +static inline void unregister_netdevice (struct net_device *dev) {unregister_netdevice_queue (dev,NULL ); } +static inline bool unregister_netdevice_queued (const struct net_device *dev) +{ +return !list_empty (&dev->unreg_list); +} +int netdev_refcnt_read (const struct net_device *dev);void free_netdev (struct net_device *dev);
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 13/16 netkit: Add xsk support for af_xdp applications
In reply to Daniel BorkmannEnable support for AF_XDP applications to operate on a netkit device. The goal is that AF_XDP applications can natively consume AF_XDP from network namespaces. The use-case from Cilium side is to support Kubernetes KubeVirt VMs through QEMU's AF_XDP backend. KubeVirt is a virtual machine management add-on for Kubernetes which aims to provide a common ground for virtualization. KubeVirt spawns the VMs inside Kubernetes Pods which reside in their own network namespace just like regular Pods.
Raw QEMU AF_XDP backend example with eth0 being a physical device with 16 queues where netkit is bound to the last queue (for multi-queue RSS context can be used if supported by the driver):
# ethtool -X eth0 start 0 equal 15
# ethtool -X eth0 start 15 equal 1 context new
# ethtool --config-ntuple eth0 flow-type ether \
src 00:00:00:00:00:00 \
src-mask ff:ff:ff:ff:ff:ff \
dst $mac dst-mask 00:00:00:00:00:00 \
proto 0 proto-mask 0xffff action 15
[ ... setup BPF/XDP prog on eth0 to steer into shared xsk map ... ]
# ip netns add foo
# ip link add numrxqueues 2 nk type netkit single
# ./pyynl/cli.py --spec ~/netlink/specs/netdev.yaml \
--do queue-create \
--json "{"ifindex": $(ifindex nk), "type": "rx", \
"lease": { "ifindex": $(ifindex eth0), \
"queue": { "type": "rx", "id": 15 } } }"
{'id': 1}
# ip link set nk netns foo
# ip netns exec foo ip link set lo up
# ip netns exec foo ip link set nk up
# ip netns exec foo qemu-system-x86_64 \
-kernel $kernel \
-drive file=${image_name},index=0,media=disk,format=raw \
-append "root=/dev/sda rw console=ttyS0" \
-cpu host \
-m $memory \
-enable-kvm \
-device virtio-net-pci,netdev=net0,mac=$mac \
-netdev af-xdp,ifname=nk,id=net0,mode=native,queues=1,start-queue=1,inhibit=on,map-path=$dir/xsks_map \
-nographicWe have tested the above against a dual-port Nvidia ConnectX-6 (mlx5) 100G NIC with successful network connectivity out of QEMU. An earlier iteration of this work was presented at LSF/MM/BPF [0] and more recently at LPC [1].
For getting to a first starting point to connect all things with KubeVirt, bind mounting the xsk map from Cilium into the VM launcher Pod which acts as a regular Kubernetes Pod while not perfect, is not a big problem given its out of reach from the application sitting inside the VM (and some of the control plane aspects are baked in the launcher Pod already), so the isolation barrier is still the VM. Eventually the goal is to have a XDP/XSK redirect extension where there is no need to have the xsk map, and the BPF program can just derive the target xsk through the queue where traffic was received on.
The exposure through netkit is because Cilium should not act as a proxy handing out xsk sockets. Existing applications expect a netdev from kernel side and should not need to rewrite just to implement against a CNI's protocol. Also, all the memory should not be accounted against Cilium but rather the application Pod itself which is consuming AF_XDP. Further, on up/downgrades we expect the data plane to being completely decoupled from the control plane; if Cilium would own the sockets that would be disruptive. Another use-case which opens up and is regularly asked from users would be to have DPDK applications on top of AF_XDP in regular Kubernetes Pods.
--- drivers/net/netkit.c | 76 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-)
drivers/net/netkit.c +75 -1
@@ -12,6 +12,7 @@#include <net/netdev_lock.h> #include <net/netdev_queues.h> #include <net/netdev_rx_queue.h> +#include <net/xdp_sock_drv.h> #include <net/netkit.h> #include <net/dst.h> #include <net/tcx.h>
@@ -235,6 +236,76 @@ static void netkit_get_stats(struct net_device *dev, stats->tx_dropped =DEV_STATS_READ (dev, tx_dropped); } +static bool netkit_xsk_supported_at_phys (const struct net_device *dev) +{ +if (!dev->netdev_ops->ndo_bpf || + !dev->netdev_ops->ndo_xdp_xmit || + !dev->netdev_ops->ndo_xsk_wakeup) +return false; +if ((dev->xdp_features &NETDEV_XDP_ACT_XSK ) !=NETDEV_XDP_ACT_XSK ) +return false; +return true; +} + +static int netkit_xsk (struct net_device *dev,struct netdev_bpf *xdp) +{ +struct netkit *nk =netkit_priv (dev); +struct netdev_bpf xdp_lower; +struct netdev_rx_queue *rxq; +struct net_device *phys; +int ret = -EBUSY ; + +switch (xdp->command) { +case XDP_SETUP_XSK_POOL : +if (nk->pair ==NETKIT_DEVICE_PAIR ) +return -EOPNOTSUPP ; +if (xdp->xsk.queue_id >= dev->real_num_rx_queues) +return -EINVAL ; + + rxq =__netif_get_rx_queue (dev, xdp->xsk.queue_id); +if (!rxq->lease) +return -EOPNOTSUPP ; + + phys = rxq->lease->dev; +if (!netkit_xsk_supported_at_phys (phys)) +return -EOPNOTSUPP ; + +memcpy (&xdp_lower, xdp,sizeof (xdp_lower)); + xdp_lower.xsk.queue_id =get_netdev_rx_queue_index (rxq->lease); +break ; +case XDP_SETUP_PROG : +return -EPERM ; +default : +return -EINVAL ; + } + +netdev_lock (phys); +if (!dev_get_min_mp_channel_count (phys)) + ret = phys->netdev_ops->ndo_bpf (phys, &xdp_lower); +netdev_unlock (phys); +return ret; +} + +static int netkit_xsk_wakeup (struct net_device *dev,u32 queue_id,u32 flags) +{ +struct netdev_rx_queue *rxq; +struct net_device *phys; + +if (queue_id >= dev->real_num_rx_queues) +return -EINVAL ; + + rxq =__netif_get_rx_queue (dev, queue_id); +if (!rxq->lease) +return -EOPNOTSUPP ; + + phys = rxq->lease->dev; +if (!netkit_xsk_supported_at_phys (phys)) +return -EOPNOTSUPP ; + +return phys->netdev_ops->ndo_xsk_wakeup (phys, +get_netdev_rx_queue_index (rxq->lease), flags); +} +static int netkit_init (struct net_device *dev) {netdev_lockdep_set_classes (dev);
@@ -255,6 +326,8 @@ static const struct net_device_ops netkit_netdev_ops = { .ndo_get_peer_dev = netkit_peer_dev, .ndo_get_stats64 = netkit_get_stats, .ndo_uninit = netkit_uninit, + .ndo_bpf = netkit_xsk, + .ndo_xsk_wakeup = netkit_xsk_wakeup, .ndo_features_check = passthru_features_check, };
@@ -381,10 +454,11 @@ static void netkit_setup(struct net_device *dev) dev->hw_enc_features = netkit_features; dev->mpls_features =NETIF_F_HW_CSUM |NETIF_F_GSO_SOFTWARE ; dev->vlan_features = dev->features & ~netkit_features_hw_vlan; - dev->needs_free_netdev = true;netif_set_tso_max_size (dev,GSO_MAX_SIZE ); + +xdp_set_features_flag (dev,NETDEV_XDP_ACT_XSK ); }static struct net *netkit_get_link_net (const struct net_device *dev)
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 14/16 selftests/net: Add bpf skb forwarding program
In reply to Daniel BorkmannFrom: David Wei <[email protected]>
Add nk_forward.bpf.c, a BPF program that forwards skbs matching some IPv6 prefix received on eth0 ifindex to a specified netkit ifindex. This will be needed by netkit container tests.
--- .../selftests/drivers/net/hw/nk_forward.bpf.c | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/hw/nk_forward.bpf.c
tools/testing/selftests/drivers/net/hw/nk_forward.bpf.c new file +49 -0
@@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/pkt_cls.h> +#include <linux/if_ether.h> +#include <linux/ipv6.h> +#include <linux/in6.h> +#include <bpf/bpf_endian.h> +#include <bpf/bpf_helpers.h> + +#define TC_ACT_OK 0 +#define ETH_P_IPV6 0x86DD + +#define ctx_ptr (field) ((void *)(long)(field)) + +#define v6_p64_equal (a, b) (a.s6_addr32[0] == b.s6_addr32[0] && \ + a.s6_addr32[1] == b.s6_addr32[1]) + +volatile __u32 netkit_ifindex; +volatile __u8 ipv6_prefix[16 ]; + +SEC("tc/ingress") +int tc_redirect_peer (struct __sk_buff *skb) +{ +void *data_end =ctx_ptr (skb->data_end); +void *data =ctx_ptr (skb->data); +struct in6_addr *peer_addr; +struct ipv6hdr *ip6h; +struct ethhdr *eth; + + peer_addr = (struct in6_addr *)ipv6_prefix; + +if (skb->protocol !=bpf_htons (ETH_P_IPV6 )) +return TC_ACT_OK ; + + eth = data; +if ((void *)(eth +1 ) > data_end) +return TC_ACT_OK ; + + ip6h = data +sizeof (struct ethhdr ); +if ((void *)(ip6h +1 ) > data_end) +return TC_ACT_OK ; + +if (!v6_p64_equal (ip6h->daddr, (*peer_addr))) +return TC_ACT_OK ; + +return bpf_redirect_peer (netkit_ifindex,0 ); +} + +char __license[] SEC("license") = "GPL";
Signature · 1 line
2.43.0
Daniel Borkmann
maintainerPATCH v8 15/16 selftests/net: Add env for container based tests
In reply to Daniel BorkmannFrom: David Wei <[email protected]>
Add an env NetDrvContEnv for container based selftests. This automates the setup of a netns, netkit pair with one inside the netns, and a BPF program that forwards skbs from the NETIF host inside the container.
Currently only netkit is used, but other virtual netdevs e.g. veth can be used too.
Expect netkit container datapath selftests to have a publicly routable IP prefix to assign to netkit in a container, such that packets will land on eth0. The BPF skb forward program will then forward such packets from the host netns to the container netns.
--- .../testing/selftests/drivers/net/README.rst | 7 + .../drivers/net/hw/lib/py/__init__.py | 7 +- .../selftests/drivers/net/lib/py/__init__.py | 7 +- .../selftests/drivers/net/lib/py/env.py | 120 ++++++++++++++++++ 4 files changed, 135 insertions(+), 6 deletions(-)
tools/testing/selftests/drivers/net/README.rst +7 -0
@@ -62,6 +62,13 @@ LOCAL_V4, LOCAL_V6, REMOTE_V4, REMOTE_V6 Local and remote endpoint IP addresses. +LOCAL_PREFIX_V4, LOCAL_PREFIX_V6 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Local IP prefix/subnet which can be used to allocate extra IP addresses (for +network name spaces behind macvlan, veth, netkit devices). DUT must be +reachable using these addresses from the endpoint. + REMOTE_TYPE ~~~~~~~~~~~
tools/testing/selftests/drivers/net/hw/lib/py/__init__.py +4 -3
@@ -3,6 +3,7 @@ """ Driver test environment (hardware-only tests). NetDrvEnv and NetDrvEpEnv are the main environment classes. +NetDrvContEnv extends NetDrvEpEnv with netkit container support. Former is for local host only tests, latter creates / connects to a remote endpoint. See NIPA wiki for more information about running and writing driver tests.
@@ -29,7 +30,7 @@ try: from net.lib.py import ksft_eq, ksft_ge, ksft_in, ksft_is, ksft_lt, \ ksft_ne, ksft_not_in, ksft_raises, ksft_true, ksft_gt, ksft_not_none from drivers.net.lib.py import GenerateTraffic, Remote, Iperf3Runner - from drivers.net.lib.py import NetDrvEnv, NetDrvEpEnv + from drivers.net.lib.py import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv __all__ = ["NetNS", "NetNSEnter", "NetdevSimDev", "EthtoolFamily", "NetdevFamily", "NetshaperFamily",
@@ -44,8 +45,8 @@ try: "ksft_eq", "ksft_ge", "ksft_in", "ksft_is", "ksft_lt", "ksft_ne", "ksft_not_in", "ksft_raises", "ksft_true", "ksft_gt", "ksft_not_none", "ksft_not_none", - "NetDrvEnv", "NetDrvEpEnv", "GenerateTraffic", "Remote", - "Iperf3Runner"] + "NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic", + "Remote", "Iperf3Runner"] except ModuleNotFoundError as e: print("Failed importing `net` library from kernel sources") print(str(e))
tools/testing/selftests/drivers/net/lib/py/__init__.py +4 -3
@@ -3,6 +3,7 @@ """ Driver test environment. NetDrvEnv and NetDrvEpEnv are the main environment classes. +NetDrvContEnv extends NetDrvEpEnv with netkit container support. Former is for local host only tests, latter creates / connects to a remote endpoint. See NIPA wiki for more information about running and writing driver tests.
@@ -43,12 +44,12 @@ try: "ksft_ne", "ksft_not_in", "ksft_raises", "ksft_true", "ksft_gt", "ksft_not_none", "ksft_not_none"] - from .env import NetDrvEnv, NetDrvEpEnv + from .env import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv from .load import GenerateTraffic, Iperf3Runner from .remote import Remote - __all__ += ["NetDrvEnv", "NetDrvEpEnv", "GenerateTraffic", "Remote", - "Iperf3Runner"] + __all__ += ["NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic", + "Remote", "Iperf3Runner"] except ModuleNotFoundError as e: print("Failed importing `net` library from kernel sources") print(str(e))
tools/testing/selftests/drivers/net/lib/py/env.py +120 -0
135 lines not on this page · raw message
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 300
Quoted 4 lines · On Thu, 29 Jan 2026 23:28:29 +0100 Daniel Borkmann wrote:
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+class NetDrvContEnv(NetDrvEpEnv): + """ + Class for an environment with a netkit pair setup for forwarding traffic + between the physical interface and a network namespace.
A little diagram would be pretty awesome to have here, and perhaps copy it to the README file too?
Quoted 12 lines
tools/testing/selftests/drivers/net/lib/py/env.py +12
-0+ """ + + def __init__(self, src_path, rxqueues=1, **kwargs): + super().__init__(src_path, **kwargs) + + self.require_ipver("6") + local_prefix = self.env.get("LOCAL_PREFIX_V6") + if not local_prefix: + raise KsftSkipEx("LOCAL_PREFIX_V6 required") + + local_prefix = local_prefix.rstrip("/64").rstrip("::").rstrip(":") + self.ipv6_prefix = f"{local_prefix}::"
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 322
Quoted 10 lines
tools/testing/selftests/drivers/net/lib/py/env.py +10
-0+ self.nk_host_ipv6 = f"{local_prefix}::2:1" + self.nk_guest_ipv6 = f"{local_prefix}::2:2" + + self.netns = None + self._nk_host_ifname = None + self._nk_guest_ifname = None + self._tc_attached = False + self._bpf_prog_pref = None + self._bpf_prog_id = None + self._init_ns_attached = False
I'm not a Python expert but the pre-init of variables that are set later on in __init__() (on all possible paths) seems like a waste of LoC
Review
Quoted 2 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")
Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLI
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 329
Quoted 4 lines
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+ all_links = ip("-d link show", json=True) + netkit_links = [link for link in all_links + if link.get('linkinfo', {}).get('info_kind') == 'netkit' + and 'UP' not in link.get('flags', [])]
And of course if you use YNL you can actually ask Netlink to echo back / return to you what interface it ended up creating. No need for the scanning..
Quoted 7 lines
tools/testing/selftests/drivers/net/lib/py/env.py +7
-0+ if len(netkit_links) != 2: + raise KsftSkipEx("Failed to create netkit pair") + + netkit_links.sort(key=lambda x: x['ifindex']) + self._nk_host_ifname = netkit_links[1]['ifname'] + self._nk_guest_ifname = netkit_links[0]['ifname'] + self.nk_host_ifindex = netkit_links[1]['ifindex']
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 10 lines
tools/testing/selftests/drivers/net/lib/py/env.py +9
-0+ self.nk_guest_ifindex = netkit_links[0]['ifindex'] + + self._setup_ns() + self._attach_bpf() + def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)
Hm, refactor class NetNSEnter or just use its enter method?
Quoted 10 lines
tools/testing/selftests/drivers/net/lib/py/env.py +10
-0+ ip(f"link set dev {self._nk_guest_ifname} netns {self.netns.name}") + ip(f"link set dev {self._nk_host_ifname} up") + ip(f"-6 addr add fe80::1/64 dev {self._nk_host_ifname} nodad") + ip(f"-6 route add {self.nk_guest_ipv6}/128 via fe80::2 dev {self._nk_host_ifname}") + + ip("link set lo up", ns=self.netns) + ip(f"link set dev {self._nk_guest_ifname} up", ns=self.netns) + ip(f"-6 addr add fe80::2/64 dev {self._nk_guest_ifname}", ns=self.netns) + ip(f"-6 addr add {self.nk_guest_ipv6}/64 dev {self._nk_guest_ifname} nodad", ns=self.netns) + ip(f"-6 route add default via fe80::1 dev {self._nk_guest_ifname}", ns=self.netns)
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 387
Quoted 10 lines
tools/testing/selftests/drivers/net/lib/py/env.py +10
-0+ + def _attach_bpf(self): + bpf_obj = self.test_dir / "nk_forward.bpf.o" + if not bpf_obj.exists(): + raise KsftSkipEx("BPF prog not found") + + cmd(f"tc filter add dev {self.ifname} ingress bpf obj {bpf_obj} sec tc/ingress direct-action") + self._tc_attached = True + + tc_info = cmd(f"tc filter show dev {self.ifname} ingress").stdout
Don't we need to also add a qdisc if there isn't one already?
Quoted 1 line
+ match =
re.search(r'pref (\d+).*nk_forward\.bpf.*id (\d+)', tc_info)
I haven't checked but TC doesn't support enough JSON and YNL doesn't support enough TC to avoid re hacks? :( :(
As I mentioned to David in person, Wei Wang also needs to use this class for PSP tests so would be awesome to detach these from the large series. I promise to prioritize the "detached" bits of these series in case you're worried about the MW.
Re: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
Quoted 8 lines · Jakub Kicinski
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 300
Quoted 4 lines · On Thu, 29 Jan 2026 23:28:29 +0100 Daniel Borkmann wrote:
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+class NetDrvContEnv(NetDrvEpEnv): + """ + Class for an environment with a netkit pair setup for forwarding traffic + between the physical interface and a network namespace.A little diagram would be pretty awesome to have here, and perhaps copy it to the README file too?
Sounds good, I'll add one. I forget how much you like ASCII diagrams.
Quoted 18 lines
Quoted 18 lines
tools/testing/selftests/drivers/net/lib/py/env.py +17
-0+ """ + + def __init__(self, src_path, rxqueues=1, **kwargs): + super().__init__(src_path, **kwargs) + + self.require_ipver("6") + local_prefix = self.env.get("LOCAL_PREFIX_V6") + if not local_prefix: + raise KsftSkipEx("LOCAL_PREFIX_V6 required") + + local_prefix = local_prefix.rstrip("/64").rstrip("::").rstrip(":") + self.ipv6_prefix = f"{local_prefix}::" + self.nk_host_ipv6 = f"{local_prefix}::2:1" + self.nk_guest_ipv6 = f"{local_prefix}::2:2" + + self.netns = None + self._nk_host_ifname = None
Quoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 322
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ self._nk_guest_ifname = None + self._tc_attached = False + self._bpf_prog_pref = None + self._bpf_prog_id = None + self._init_ns_attached = FalseI'm not a Python expert but the pre-init of variables that are set later on in
__init__()(on all possible paths) seems like a waste of LoC
It's so that exceptions thrown -> __del__() don't complain about undefined vars.
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLI
Hmm, I tried looking into this already. My understanding is that this can be done in C but not currently in YNL, not without adding a spec.
Quoted 22 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 329
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+ all_links = ip("-d link show", json=True) + netkit_links = [link for link in all_links + if link.get('linkinfo', {}).get('info_kind') == 'netkit' + and 'UP' not in link.get('flags', [])]And of course if you use YNL you can actually ask Netlink to echo back / return to you what interface it ended up creating. No need for the scanning..
Quoted 11 lines
tools/testing/selftests/drivers/net/lib/py/env.py +11
-0+ if len(netkit_links) != 2: + raise KsftSkipEx("Failed to create netkit pair") + + netkit_links.sort(key=lambda x: x['ifindex']) + self._nk_host_ifname = netkit_links[1]['ifname'] + self._nk_guest_ifname = netkit_links[0]['ifname'] + self.nk_host_ifindex = netkit_links[1]['ifindex'] + self.nk_guest_ifindex = netkit_links[0]['ifindex'] + + self._setup_ns() + self._attach_bpf()
Quoted 7 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)Hm, refactor class NetNSEnter or just use its enter method?
I did consider this but I didn't feel like it fit the very specific purpose of NetNSEnter. I needed the init ns to have a valid nsid from within the test netns, which I didn't think many other tests would need.
Quoted 15 lines
Quoted 15 lines
tools/testing/selftests/drivers/net/lib/py/env.py +14
-0+ ip(f"link set dev {self._nk_guest_ifname} netns {self.netns.name}") + ip(f"link set dev {self._nk_host_ifname} up") + ip(f"-6 addr add fe80::1/64 dev {self._nk_host_ifname} nodad") + ip(f"-6 route add {self.nk_guest_ipv6}/128 via fe80::2 dev {self._nk_host_ifname}") + + ip("link set lo up", ns=self.netns) + ip(f"link set dev {self._nk_guest_ifname} up", ns=self.netns) + ip(f"-6 addr add fe80::2/64 dev {self._nk_guest_ifname}", ns=self.netns) + ip(f"-6 addr add {self.nk_guest_ipv6}/64 dev {self._nk_guest_ifname} nodad", ns=self.netns) + ip(f"-6 route add default via fe80::1 dev {self._nk_guest_ifname}", ns=self.netns) + + def _attach_bpf(self): + bpf_obj = self.test_dir / "nk_forward.bpf.o" + if not bpf_obj.exists():
Quoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 387
Quoted 6 lines
tools/testing/selftests/drivers/net/lib/py/env.py +6
-0+ raise KsftSkipEx("BPF prog not found") + + cmd(f"tc filter add dev {self.ifname} ingress bpf obj {bpf_obj} sec tc/ingress direct-action") + self._tc_attached = True + + tc_info = cmd(f"tc filter show dev {self.ifname} ingress").stdoutDon't we need to also add a qdisc if there isn't one already?
Sorry, why would qdisc be needed? The bpf prog attaches to tc. I don't know anything about qdisc, though.
Quoted 5 lines
Quoted 2 lines
+ match =
re.search(r'pref (\d+).*nk_forward\.bpf.*id (\d+)', tc_info)I haven't checked but TC doesn't support enough JSON and YNL doesn't support enough TC to avoid re hacks? :( :(
Yeah the old TC tool doesn't give me this info programmatically :(
Borkmann mentioned TC is outdated/unmaintained. I did use tcx via a C loader before, but that was removed in an earlier iteration.
Quoted 6 lines
As I mentioned to David in person, Wei Wang also needs to use this class for PSP tests so would be awesome to detach these from the large series. I promise to prioritize the "detached" bits of these series in case you're worried about the MW.
I'll move the env, basic ping test out. Then keep the queue lease tests, adding the error paths you requested, in this series.
Jakub Kicinski
maintainerRe: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
Quoted 6 lines · On Mon, 2 Feb 2026 07:53:34 +0900 David Wei wrote:
Quoted 2 lines · Jakub Kicinski
I'm not a Python expert but the pre-init of variables that are set later on in
__init__()(on all possible paths) seems like a waste of LoCIt's so that exceptions thrown ->
__del__()don't complain about undefined vars.
I see. I guess that's cleaner than hasattr(self, "xyz"), fair.
Quoted 8 lines
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLIHmm, I tried looking into this already. My understanding is that this can be done in C but not currently in YNL, not without adding a spec.
I thought the only thing C can do that Python can't is deciding nested extacks.. Ah, of course.. I missed that patch 10 is not updating the YAML spec. That needs to be fixed, we want YAML to stay in sync for the families that are already YAML-ized.
Quoted 11 lines
Quoted 7 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)Hm, refactor class NetNSEnter or just use its enter method?
I did consider this but I didn't feel like it fit the very specific purpose of NetNSEnter. I needed the init ns to have a valid nsid from within the test netns, which I didn't think many other tests would need.
What about the "ip netns attach init 1" ?
that's the main thing that gave me pause TBH
Why is it needed?
Quoted 4 lines
Quoted 1 line
Don't we need to also add a qdisc if there isn't one already?
Sorry, why would qdisc be needed? The bpf prog attaches to tc. I don't know anything about qdisc, though.
There's a new way of attaching qdisc-less (TCX) but IIUC here you're attaching in the old way. So you must have a parent object. The ingress / clsact qdisc isn't really for queuing packets, it's just a fake qdisc that lets you attach TC filters to an interface. Machines at Meta have a clsact qdisc installed by init, but that's far from normal (try to run this command in vng, you'll see)
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
+ match =
re.search(r'pref (\d+).*nk_forward\.bpf.*id (\d+)', tc_info)I haven't checked but TC doesn't support enough JSON and YNL doesn't support enough TC to avoid re hacks? :( :(
Yeah the old TC tool doesn't give me this info programmatically :(
Borkmann mentioned TC is outdated/unmaintained. I did use tcx via a C loader before, but that was removed in an earlier iteration.
Not sure what you're saying. tc bpf CLI supports JSON output since:
commit 3175bca7182b0867c6e9a3d5d1551fdecf70118c
Author: Davide Caratti <[email protected]> Date: Thu Apr 30 20:03:30 2020 +0200
tc: full JSON support for 'bpf' filter
Re: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
Quoted 16 lines · Jakub Kicinski
Quoted 6 lines · On Mon, 2 Feb 2026 07:53:34 +0900 David Wei wrote:
Quoted 2 lines · Jakub Kicinski
I'm not a Python expert but the pre-init of variables that are set later on in
__init__()(on all possible paths) seems like a waste of LoCIt's so that exceptions thrown ->
__del__()don't complain about undefined vars.I see. I guess that's cleaner than
hasattr(self, "xyz"), fair.Quoted 6 lines
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLI
Quoted 7 lines
Quoted 2 lines
Hmm, I tried looking into this already. My understanding is that this can be done in C but not currently in YNL, not without adding a spec.
I thought the only thing C can do that Python can't is deciding nested extacks.. Ah, of course.. I missed that patch 10 is not updating the YAML spec. That needs to be fixed, we want YAML to stay in sync for the families that are already YAML-ized.
Sorry, my mistake. It is possible with rt-link.yaml:
ynl = RtnlFamily()
ynl.newlink(...)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 6 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)Hm, refactor class NetNSEnter or just use its enter method?
Quoted 7 lines
Quoted 3 lines
I did consider this but I didn't feel like it fit the very specific purpose of NetNSEnter. I needed the init ns to have a valid nsid from within the test netns, which I didn't think many other tests would need.
What about the "ip netns attach init 1" ?
that's the main thing that gave me pause TBH
Why is it needed?
The bind-queue lease attribute takes the ifindex/queue/netns-id of the source phys netdev to lease a queue from. Once a netlink has been moved into a netns, it doesn't have access to the source phys netdev in the init ns anymore. This is why initially I did the bind-queue before moving one of the netkit peers into a netns.
For me to be able to do a bind-queue op inside a netns, I use ip netns attach to make the init ns available with a nsid inside of a netns.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Don't we need to also add a qdisc if there isn't one already?
Sorry, why would qdisc be needed? The bpf prog attaches to tc. I don't know anything about qdisc, though.
Quoted 6 lines
There's a new way of attaching qdisc-less (TCX) but IIUC here you're attaching in the old way. So you must have a parent object. The ingress / clsact qdisc isn't really for queuing packets, it's just a fake qdisc that lets you attach TC filters to an interface. Machines at Meta have a clsact qdisc installed by init, but that's far from normal (try to run this command in vng, you'll see)
Thanks for the explanation. I'll fix this by adding a qdisc if one doesn't exist.
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
+ match =
re.search(r'pref (\d+).*nk_forward\.bpf.*id (\d+)', tc_info)I haven't checked but TC doesn't support enough JSON and YNL doesn't support enough TC to avoid re hacks? :( :(
Yeah the old TC tool doesn't give me this info programmatically :(
Borkmann mentioned TC is outdated/unmaintained. I did use tcx via a C loader before, but that was removed in an earlier iteration.
Quoted 7 lines
Not sure what you're saying. tc bpf CLI supports JSON output since:
commit
3175bca7182b0867c6e9a3d5d1551fdecf70118cAuthor: Davide Caratti <[email protected]> Date: Thu Apr 30 20:03:30 2020 +0200tc: full JSON support for 'bpf' filter
Yes this works, I'll use it instead of regex.
Re: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
In reply to David WeiQuoted 110 lines · David Wei
Quoted 8 lines · Jakub Kicinski
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 300
Quoted 4 lines · On Thu, 29 Jan 2026 23:28:29 +0100 Daniel Borkmann wrote:
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+class NetDrvContEnv(NetDrvEpEnv): + """ + Class for an environment with a netkit pair setup for forwarding traffic + between the physical interface and a network namespace.A little diagram would be pretty awesome to have here, and perhaps copy it to the README file too?
Sounds good, I'll add one. I forget how much you like ASCII diagrams.
Quoted 18 lines
Quoted 18 lines
tools/testing/selftests/drivers/net/lib/py/env.py +17
-0+ """ + + def __init__(self, src_path, rxqueues=1, **kwargs): + super().__init__(src_path, **kwargs) + + self.require_ipver("6") + local_prefix = self.env.get("LOCAL_PREFIX_V6") + if not local_prefix: + raise KsftSkipEx("LOCAL_PREFIX_V6 required") + + local_prefix = local_prefix.rstrip("/64").rstrip("::").rstrip(":") + self.ipv6_prefix = f"{local_prefix}::" + self.nk_host_ipv6 = f"{local_prefix}::2:1" + self.nk_guest_ipv6 = f"{local_prefix}::2:2" + + self.netns = None + self._nk_host_ifname = NoneQuoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 322
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ self._nk_guest_ifname = None + self._tc_attached = False + self._bpf_prog_pref = None + self._bpf_prog_id = None + self._init_ns_attached = FalseI'm not a Python expert but the pre-init of variables that are set later on in
__init__()(on all possible paths) seems like a waste of LoCIt's so that exceptions thrown ->
__del__()don't complain about undefined vars.Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLIHmm, I tried looking into this already. My understanding is that this can be done in C but not currently in YNL, not without adding a spec.
Quoted 22 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 329
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+ all_links = ip("-d link show", json=True) + netkit_links = [link for link in all_links + if link.get('linkinfo', {}).get('info_kind') == 'netkit' + and 'UP' not in link.get('flags', [])]And of course if you use YNL you can actually ask Netlink to echo back / return to you what interface it ended up creating. No need for the scanning..
Quoted 11 lines
tools/testing/selftests/drivers/net/lib/py/env.py +11
-0+ if len(netkit_links) != 2: + raise KsftSkipEx("Failed to create netkit pair") + + netkit_links.sort(key=lambda x: x['ifindex']) + self._nk_host_ifname = netkit_links[1]['ifname'] + self._nk_guest_ifname = netkit_links[0]['ifname'] + self.nk_host_ifindex = netkit_links[1]['ifindex'] + self.nk_guest_ifindex = netkit_links[0]['ifindex'] + + self._setup_ns() + self._attach_bpf()Quoted 7 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)Hm, refactor class NetNSEnter or just use its enter method?
I did consider this but I didn't feel like it fit the very specific purpose of NetNSEnter. I needed the init ns to have a valid nsid from within the test netns, which I didn't think many other tests would need.
Quoted 15 lines
Quoted 15 lines
tools/testing/selftests/drivers/net/lib/py/env.py +14
-0+ ip(f"link set dev {self._nk_guest_ifname} netns {self.netns.name}") + ip(f"link set dev {self._nk_host_ifname} up") + ip(f"-6 addr add fe80::1/64 dev {self._nk_host_ifname} nodad") + ip(f"-6 route add {self.nk_guest_ipv6}/128 via fe80::2 dev {self._nk_host_ifname}") + + ip("link set lo up", ns=self.netns) + ip(f"link set dev {self._nk_guest_ifname} up", ns=self.netns) + ip(f"-6 addr add fe80::2/64 dev {self._nk_guest_ifname}", ns=self.netns) + ip(f"-6 addr add {self.nk_guest_ipv6}/64 dev {self._nk_guest_ifname} nodad", ns=self.netns) + ip(f"-6 route add default via fe80::1 dev {self._nk_guest_ifname}", ns=self.netns) + + def _attach_bpf(self): + bpf_obj = self.test_dir / "nk_forward.bpf.o" + if not bpf_obj.exists():Quoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 387
Quoted 6 lines
tools/testing/selftests/drivers/net/lib/py/env.py +6
-0+ raise KsftSkipEx("BPF prog not found") + + cmd(f"tc filter add dev {self.ifname} ingress bpf obj {bpf_obj} sec tc/ingress direct-action") + self._tc_attached = True + + tc_info = cmd(f"tc filter show dev {self.ifname} ingress").stdoutDon't we need to also add a qdisc if there isn't one already?
Quoted 2 lines
Sorry, why would qdisc be needed? The bpf prog attaches to tc. I don't know anything about qdisc, though.
The ingress qdisc is the attachment point. I think we need CONFIG_NET_SCH_INGRESS here too?
Best,
Bobby
Re: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
Quoted 111 lines · Bobby Eshleman
Quoted 110 lines · David Wei
Quoted 8 lines · Jakub Kicinski
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 300
Quoted 4 lines · On Thu, 29 Jan 2026 23:28:29 +0100 Daniel Borkmann wrote:
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+class NetDrvContEnv(NetDrvEpEnv): + """ + Class for an environment with a netkit pair setup for forwarding traffic + between the physical interface and a network namespace.A little diagram would be pretty awesome to have here, and perhaps copy it to the README file too?
Sounds good, I'll add one. I forget how much you like ASCII diagrams.
Quoted 18 lines
Quoted 18 lines
tools/testing/selftests/drivers/net/lib/py/env.py +17
-0+ """ + + def __init__(self, src_path, rxqueues=1, **kwargs): + super().__init__(src_path, **kwargs) + + self.require_ipver("6") + local_prefix = self.env.get("LOCAL_PREFIX_V6") + if not local_prefix: + raise KsftSkipEx("LOCAL_PREFIX_V6 required") + + local_prefix = local_prefix.rstrip("/64").rstrip("::").rstrip(":") + self.ipv6_prefix = f"{local_prefix}::" + self.nk_host_ipv6 = f"{local_prefix}::2:1" + self.nk_guest_ipv6 = f"{local_prefix}::2:2" + + self.netns = None + self._nk_host_ifname = NoneQuoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 322
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ self._nk_guest_ifname = None + self._tc_attached = False + self._bpf_prog_pref = None + self._bpf_prog_id = None + self._init_ns_attached = FalseI'm not a Python expert but the pre-init of variables that are set later on in
__init__()(on all possible paths) seems like a waste of LoCIt's so that exceptions thrown ->
__del__()don't complain about undefined vars.Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLIHmm, I tried looking into this already. My understanding is that this can be done in C but not currently in YNL, not without adding a spec.
Quoted 22 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 329
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+ all_links = ip("-d link show", json=True) + netkit_links = [link for link in all_links + if link.get('linkinfo', {}).get('info_kind') == 'netkit' + and 'UP' not in link.get('flags', [])]And of course if you use YNL you can actually ask Netlink to echo back / return to you what interface it ended up creating. No need for the scanning..
Quoted 11 lines
tools/testing/selftests/drivers/net/lib/py/env.py +11
-0+ if len(netkit_links) != 2: + raise KsftSkipEx("Failed to create netkit pair") + + netkit_links.sort(key=lambda x: x['ifindex']) + self._nk_host_ifname = netkit_links[1]['ifname'] + self._nk_guest_ifname = netkit_links[0]['ifname'] + self.nk_host_ifindex = netkit_links[1]['ifindex'] + self.nk_guest_ifindex = netkit_links[0]['ifindex'] + + self._setup_ns() + self._attach_bpf()Quoted 7 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)Hm, refactor class NetNSEnter or just use its enter method?
I did consider this but I didn't feel like it fit the very specific purpose of NetNSEnter. I needed the init ns to have a valid nsid from within the test netns, which I didn't think many other tests would need.
Quoted 23 lines
Quoted 11 lines
tools/testing/selftests/drivers/net/lib/py/env.py +10
-0+ ip(f"link set dev {self._nk_guest_ifname} netns {self.netns.name}") + ip(f"link set dev {self._nk_host_ifname} up") + ip(f"-6 addr add fe80::1/64 dev {self._nk_host_ifname} nodad") + ip(f"-6 route add {self.nk_guest_ipv6}/128 via fe80::2 dev {self._nk_host_ifname}") + + ip("link set lo up", ns=self.netns) + ip(f"link set dev {self._nk_guest_ifname} up", ns=self.netns) + ip(f"-6 addr add fe80::2/64 dev {self._nk_guest_ifname}", ns=self.netns) + ip(f"-6 addr add {self.nk_guest_ipv6}/64 dev {self._nk_guest_ifname} nodad", ns=self.netns) + ip(f"-6 route add default via fe80::1 dev {self._nk_guest_ifname}", ns=self.netns)tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 387
Quoted 10 lines
tools/testing/selftests/drivers/net/lib/py/env.py +10
-0+ + def _attach_bpf(self): + bpf_obj = self.test_dir / "nk_forward.bpf.o" + if not bpf_obj.exists(): + raise KsftSkipEx("BPF prog not found") + + cmd(f"tc filter add dev {self.ifname} ingress bpf obj {bpf_obj} sec tc/ingress direct-action") + self._tc_attached = True + + tc_info = cmd(f"tc filter show dev {self.ifname} ingress").stdoutDon't we need to also add a qdisc if there isn't one already?
Quoted 5 lines
Quoted 2 lines
Sorry, why would qdisc be needed? The bpf prog attaches to tc. I don't know anything about qdisc, though.
The ingress qdisc is the attachment point. I think we need
CONFIG_NET_SCH_INGRESShere too?
Sorry for the config stuff I'm replying about, I ran into needing these too for this:
CONFIG_NET_CLS_BPF=yCONFIG_BPF_SYSCALL=y
Re: [PATCH net-next v8 15/16] selftests/net: Add env for container based tests
Quoted 115 lines · Bobby Eshleman
Quoted 114 lines · Bobby Eshleman
Quoted 112 lines · David Wei
Quoted 8 lines · Jakub Kicinski
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 300
Quoted 4 lines · On Thu, 29 Jan 2026 23:28:29 +0100 Daniel Borkmann wrote:
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+class NetDrvContEnv(NetDrvEpEnv): + """ + Class for an environment with a netkit pair setup for forwarding traffic + between the physical interface and a network namespace.A little diagram would be pretty awesome to have here, and perhaps copy it to the README file too?
Sounds good, I'll add one. I forget how much you like ASCII diagrams.
Quoted 18 lines
Quoted 18 lines
tools/testing/selftests/drivers/net/lib/py/env.py +17
-0+ """ + + def __init__(self, src_path, rxqueues=1, **kwargs): + super().__init__(src_path, **kwargs) + + self.require_ipver("6") + local_prefix = self.env.get("LOCAL_PREFIX_V6") + if not local_prefix: + raise KsftSkipEx("LOCAL_PREFIX_V6 required") + + local_prefix = local_prefix.rstrip("/64").rstrip("::").rstrip(":") + self.ipv6_prefix = f"{local_prefix}::" + self.nk_host_ipv6 = f"{local_prefix}::2:1" + self.nk_guest_ipv6 = f"{local_prefix}::2:2" + + self.netns = None + self._nk_host_ifname = NoneQuoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 322
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ self._nk_guest_ifname = None + self._tc_attached = False + self._bpf_prog_pref = None + self._bpf_prog_id = None + self._init_ns_attached = FalseI'm not a Python expert but the pre-init of variables that are set later on in
__init__()(on all possible paths) seems like a waste of LoCIt's so that exceptions thrown ->
__del__()don't complain about undefined vars.Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ + ip(f"link add type netkit mode l2 forward peer forward numrxqueues {rxqueues}")Would we be able to do this with YNL?
no big deal but always nice to avoid the dependency on very latest CLIHmm, I tried looking into this already. My understanding is that this can be done in C but not currently in YNL, not without adding a spec.
Quoted 22 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 329
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +4
-0+ all_links = ip("-d link show", json=True) + netkit_links = [link for link in all_links + if link.get('linkinfo', {}).get('info_kind') == 'netkit' + and 'UP' not in link.get('flags', [])]And of course if you use YNL you can actually ask Netlink to echo back / return to you what interface it ended up creating. No need for the scanning..
Quoted 11 lines
tools/testing/selftests/drivers/net/lib/py/env.py +11
-0+ if len(netkit_links) != 2: + raise KsftSkipEx("Failed to create netkit pair") + + netkit_links.sort(key=lambda x: x['ifindex']) + self._nk_host_ifname = netkit_links[1]['ifname'] + self._nk_guest_ifname = netkit_links[0]['ifname'] + self.nk_host_ifindex = netkit_links[1]['ifindex'] + self.nk_guest_ifindex = netkit_links[0]['ifindex'] + + self._setup_ns() + self._attach_bpf()Quoted 7 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 367
Quoted 5 lines
tools/testing/selftests/drivers/net/lib/py/env.py +5
-0+ def _setup_ns(self): + self.netns = NetNS() + cmd("ip netns attach init 1") + self._init_ns_attached = True + ip("netns set init 0", ns=self.netns)Hm, refactor class NetNSEnter or just use its enter method?
I did consider this but I didn't feel like it fit the very specific purpose of NetNSEnter. I needed the init ns to have a valid nsid from within the test netns, which I didn't think many other tests would need.
Quoted 15 lines
Quoted 15 lines
tools/testing/selftests/drivers/net/lib/py/env.py +14
-0+ ip(f"link set dev {self._nk_guest_ifname} netns {self.netns.name}") + ip(f"link set dev {self._nk_host_ifname} up") + ip(f"-6 addr add fe80::1/64 dev {self._nk_host_ifname} nodad") + ip(f"-6 route add {self.nk_guest_ipv6}/128 via fe80::2 dev {self._nk_host_ifname}") + + ip("link set lo up", ns=self.netns) + ip(f"link set dev {self._nk_guest_ifname} up", ns=self.netns) + ip(f"-6 addr add fe80::2/64 dev {self._nk_guest_ifname}", ns=self.netns) + ip(f"-6 addr add {self.nk_guest_ipv6}/64 dev {self._nk_guest_ifname} nodad", ns=self.netns) + ip(f"-6 route add default via fe80::1 dev {self._nk_guest_ifname}", ns=self.netns) + + def _attach_bpf(self): + bpf_obj = self.test_dir / "nk_forward.bpf.o" + if not bpf_obj.exists():Quoted 8 lines
tools/testing/selftests/drivers/net/lib/py/env.py class NetDrvEpEnv(NetDrvEnvBase): · line 387
Quoted 6 lines
tools/testing/selftests/drivers/net/lib/py/env.py +6
-0+ raise KsftSkipEx("BPF prog not found") + + cmd(f"tc filter add dev {self.ifname} ingress bpf obj {bpf_obj} sec tc/ingress direct-action") + self._tc_attached = True + + tc_info = cmd(f"tc filter show dev {self.ifname} ingress").stdoutDon't we need to also add a qdisc if there isn't one already?
Sorry, why would qdisc be needed? The bpf prog attaches to tc. I don't know anything about qdisc, though.
Quoted 8 lines
Quoted 2 lines
The ingress qdisc is the attachment point. I think we need
CONFIG_NET_SCH_INGRESShere too?Sorry for the config stuff I'm replying about, I ran into needing these too for this:
CONFIG_NET_CLS_BPF=yCONFIG_BPF_SYSCALL=y
Thanks Bobby, I'll add these to the kconfig.
Daniel Borkmann
maintainerPATCH v8 16/16 selftests/net: Add netkit container tests
In reply to Daniel BorkmannFrom: David Wei <[email protected]>
Add two tests using NetDrvContEnv. One basic test that sets up a netkit pair, with one end in a netns. Use LOCAL_PREFIX_V6 and nk_forward BPF program to ping from a remote host to the netkit in netns.
Second is a selftest for netkit queue leasing, using io_uring zero copy test binary inside of a netns with netkit. This checks that memory providers can be bound against virtual queues in a netkit within a netns that are leasing from a physical netdev in the default netns.
--- .../testing/selftests/drivers/net/hw/Makefile | 2 + .../selftests/drivers/net/hw/nk_netns.py | 23 +++++ .../selftests/drivers/net/hw/nk_qlease.py | 90 +++++++++++++++++++ 3 files changed, 115 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/hw/nk_netns.py create mode 100755 tools/testing/selftests/drivers/net/hw/nk_qlease.py
tools/testing/selftests/drivers/net/hw/Makefile +2 -0
@@ -32,6 +32,8 @@ TEST_PROGS = \ irq.py \ loopback.sh \ nic_timestamp.py \ + nk_netns.py \ + nk_qlease.py \ pp_alloc_fail.py \ rss_api.py \ rss_ctx.py \
tools/testing/selftests/drivers/net/hw/nk_netns.py new file +23 -0
@@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +from lib.py import ksft_run, ksft_exit +from lib.py import NetDrvContEnv +from lib.py import cmd + + +def test_ping(cfg) -> None: + cfg.require_ipver("6") + + cmd(f"ping -c 1 -W5 {cfg.nk_guest_ipv6}", host=cfg.remote) + cmd(f"ping -c 1 -W5 {cfg.remote_addr_v['6']}", ns=cfg.netns) + + +def main() -> None: + with NetDrvContEnv(__file__) as cfg: + ksft_run([test_ping], args=(cfg,)) + ksft_exit() + + +if __name__ == "__main__": + main()
tools/testing/selftests/drivers/net/hw/nk_qlease.py new file +90 -0
90 lines not on this page · raw message
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next v8 16/16] selftests/net: Add netkit container tests
Quoted 8 lines · On Thu, 29 Jan 2026 23:28:30 +0100 Daniel Borkmann wrote:
Add two tests using NetDrvContEnv. One basic test that sets up a netkit pair, with one end in a netns. Use
LOCAL_PREFIX_V6andnk_forwardBPF program to ping from a remote host to the netkit in netns.Second is a selftest for netkit queue leasing, using
io_uringzero copy test binary inside of a netns with netkit. This checks that memory providers can be bound against virtual queues in a netkit within a netns that are leasing from a physical netdev in the default netns.
We should cover the uAPI and the tricky part of the handling please.
So off the top of my head main bits:
- the expected attributes on lessor and lessee via Netlink
- destroying the netkit with MP attached
- checking the state via YNL,
- attaching MP from the main dev to make sure we didn't leave anything behind
- destroying phys dev while netkit exists (prolly netdevsim :S)
- making sure XDP can't be attached on phys dev when MP is installed via netkit
... whatever else you found to be tricky in the implementation.
Re: [PATCH net-next v8 16/16] selftests/net: Add netkit container tests
Quoted 22 lines · Jakub Kicinski
Quoted 8 lines · On Thu, 29 Jan 2026 23:28:30 +0100 Daniel Borkmann wrote:
Add two tests using NetDrvContEnv. One basic test that sets up a netkit pair, with one end in a netns. Use
LOCAL_PREFIX_V6andnk_forwardBPF program to ping from a remote host to the netkit in netns.Second is a selftest for netkit queue leasing, using
io_uringzero copy test binary inside of a netns with netkit. This checks that memory providers can be bound against virtual queues in a netkit within a netns that are leasing from a physical netdev in the default netns.We should cover the uAPI and the tricky part of the handling please.
So off the top of my head main bits:
- the expected attributes on lessor and lessee via Netlink
- destroying the netkit with MP attached
- checking the state via YNL,
- attaching MP from the main dev to make sure we didn't leave anything behind
- destroying phys dev while netkit exists (prolly netdevsim :S)
- making sure XDP can't be attached on phys dev when MP is installed via netkit
Quoted 1 line
... whatever else you found to be tricky in the implementation.
Sounds good. I'll add these test cases and more.
Re: [PATCH net-next v8 16/16] selftests/net: Add netkit container tests
In reply to Daniel BorkmannQuoted 26 lines · Daniel Borkmann
From: David Wei <[email protected]>
Add two tests using NetDrvContEnv. One basic test that sets up a netkit pair, with one end in a netns. Use
LOCAL_PREFIX_V6andnk_forwardBPF program to ping from a remote host to the netkit in netns.Second is a selftest for netkit queue leasing, using
io_uringzero copy test binary inside of a netns with netkit. This checks that memory providers can be bound against virtual queues in a netkit within a netns that are leasing from a physical netdev in the default netns.Signed-off-by David Wei <[email protected]>Signed-off-by Daniel Borkmann <[email protected]>--- .../testing/selftests/drivers/net/hw/Makefile | 2 + .../selftests/drivers/net/hw/nk_netns.py | 23 +++++ .../selftests/drivers/net/hw/nk_qlease.py | 90 +++++++++++++++++++ 3 files changed, 115 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/hw/nk_netns.py create mode 100755 tools/testing/selftests/drivers/net/hw/nk_qlease.pytools/testing/selftests/drivers/net/hw/Makefile +0
-0
tools/testing/selftests/drivers/net/hw/Makefile TEST_PROGS = \ · line 39
Quoted 8 lines
tools/testing/selftests/drivers/net/hw/Makefile +2
-0@@ -32,6 +32,8 @@ TEST_PROGS = \ irq.py \ loopback.sh \ nic_timestamp.py \ + nk_netns.py \ + nk_qlease.py \ pp_alloc_fail.py \ rss_api.py \ rss_ctx.py \
I think we might also need something like:
tools/testing/selftests/drivers/net/hw/config +1 -0
@@ -5,6 +5,7 @@ CONFIG_FUNCTION_ERROR_INJECTION=y CONFIG_IO_URING=y CONFIG_IPV6=y CONFIG_IPV6_GRE=y +CONFIG_NETKIT=y CONFIG_NET_IPGRE=y CONFIG_NET_IPGRE_DEMUX=y CONFIG_UDMABUF=y
Best,
Bobby