netgpu: networking between NIC and GPU/CPU.
Subsystems (14)
RFC PATCH v2 2/21 mm/memory_hotplug: add {add|release}_memory_pages
In reply to Jonathan LemonThese calls allows creation of system pages at a specific physical address, which is useful for creating dummy backing pages which correspond to unaddressable external memory at specific locations.
__add_memory_pages() adds the requested page range to /proc/iomem and verifies that there are no overlaps. Once this succeeds, then the page section is initialized, which may overlap with a prior request, since section sizes are large, so ignore the latter overlap.
--- include/linux/memory_hotplug.h | 4 +++ mm/memory_hotplug.c | 65 ++++++++++++++++++++++++++++++++-- 2 files changed, 67 insertions(+), 2 deletions(-)
include/linux/memory_hotplug.h +4 -0
@@ -138,6 +138,10 @@ extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,extern int __add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages,struct mhp_params *params); +struct resource *add_memory_pages (int nid,u64 start,u64 size, +struct mhp_params *params); +void release_memory_pages (struct resource *res); +#ifndef CONFIG_ARCH_HAS_ADD_PAGES static inline int add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages,struct mhp_params *params)
mm/memory_hotplug.c +63 -2
77 lines not on this page · raw message
Signature · 1 line
2.24.1
v2 cover netgpu: networking between NIC and GPU/CPU.
From: Jonathan Lemon <[email protected]>
[ RESENDING, as apparently the initial submission was lost ]
This series is a working RFC proof-of-concept that implements DMA zero-copy between the NIC and a GPU device for the data path, while keeping the protocol processing on the host CPU.
This also works for zero-copy send/recv to host (CPU) memory.
Current limitations:
- mlx5 only, header splitting is at a fixed offset.
- currently only TCP protocol delivery is performed.
- TX completion notification is planned, but not in this patchset.
- not compatible with xsk (re-uses same datastructures)
- not compatible with bpf payload inspection
Changes since v1:
- user api restructured to provide more flexibility.
- iommu issues resolved
- performance issues fixed.
- nvidia bits are built as an external module.
The next section provides a brief overview of how things work.
A transport context (aka RSS object) is created for a device, which acts as a container for a group of queues. Queues are opened on a context, these correcpond to hardware queues on the NIC. There exists the ability to request a sepcific numbered queue, or "any" queue. Only RX queues are needed, the standard TX queues are used for packet transmission.
Memory regions which participate in zero-copy transmission (either send or recieve) are registered with a memory object, and then a specific region can be attached to a context for usage. This way, multiple contexts can share the same memory regions. These areas can be used as either RX packet buffers or TX data areas (or both). The memory can come from either malloc/mmap or cudaMalloc(). The latter call provides a handle to the userspace application, but the memory region is only accessible to the GPU.
A socket is created and registered with the context, which sets SOCK_ZEROCOPY, also creates a per-socket queue for recieving zero-copy data.
Asymmetrical data paths are possible (zc TX, normal RX), and vice versa, but the curreent PoC sets things up for symmetrical transport. The application needs to provide the RX buffers to the ifq fill queue, similar to AF_XDP.
Once things are set up, data is sent to the network with sendmsg(). The iovecs provided contain an address in the region previously registered. The normal protocol stack processing constructs the packet, but the data is not touched by the stack. In this phase, the application is not notified when the protocol processing is complete and the data area is safe to modify again.
For RX, packets undergo the usual protocol processing and are delivered up to the socket receive queue. At this point, the skb data fragments are delivered to the application as iovecs through an AF_XDP style queue which belongs to the socket. The application can poll for readability, but does not use read() to receive the data.
The initial application used is iperf3, a modified version with the userspace library corresponding to this patch is available at:
https://github.com/jlemon/iperf
https://github.com/jlemon/netgpu
Running "iperf3 -s -z --dport 8888" (host memory) on a 12Gbps link:
11.4 Gbit/sec receive
10.8 Gbit/sec transmit
Running "iperf3 -s -z --dport 8888 --gpu" on a 25Gbps link:
23.4 Gbit/sec receive
23.8 Gbit/sec transmit
For the GPU runs, the Intel PCI monitoring tools were used to confirm that the host PCI bus was mostly idle.
Patch series:
1,4 : cleanup & extension 2,3 : extend mm, allowing allocation of specific memory regions 5,6,7 : add include support files 8,9,11 : skbuff core changes for handling zc frags 10,21 : netgpu and nvidia code 12-15 : changes to connect up netgpu code 16-20 : mlx5 driver changes
Comments eagerly solicited.
--
Jonathan
Jonathan Lemon (21):
linux/log2.h: enclose macro arg in parens mm/memory_hotplug: add {add|release}_memory_pages
mm: Allow DMA mapping of pages which are not online kernel/user: export free_uid
uapi/misc: add shqueue.h for shared queues
include: add netgpu UAPI and kernel definitions
netdevice: add SETUP_NETGPU to the netdev_bpf structure
skbuff: add a zc_netgpu bitflag
core/skbuff: use skb_zdata for testing whether skb is zerocopy
netgpu: add network/gpu/host dma module
core/skbuff: add page recycling logic for netgpu pages
lib: have __zerocopy_sg_from_iter get netgpu pages for a sk net/tcp: Pad TCP options out to a fixed size for netgpu net/tcp: add netgpu ioctl setting up zero copy RX queues net/tcp: add MSG_NETDMA flag for sendmsg()
mlx5: remove the umem parameter from mlx5e_open_channel
mlx5e: add header split ability
mlx5e: add netgpu entries to mlx5 structures
mlx5e: add the netgpu driver functions
mlx5e: hook up the netgpu functions
netgpu/nvidia: add Nvidia plugin for netgpu
drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/netgpu/Kconfig | 14 + drivers/misc/netgpu/Makefile | 6 + drivers/misc/netgpu/netgpu_host.c | 284 ++++ drivers/misc/netgpu/netgpu_main.c | 1215 +++++++++++++++++ drivers/misc/netgpu/netgpu_mem.c | 351 +++++ drivers/misc/netgpu/netgpu_priv.h | 88 ++ drivers/misc/netgpu/netgpu_stub.c | 166 +++ drivers/misc/netgpu/netgpu_stub.h | 19 + drivers/misc/netgpu/nvidia/Kbuild | 9 + drivers/misc/netgpu/nvidia/Kconfig | 10 + drivers/misc/netgpu/nvidia/netgpu_cuda.c | 416 ++++++ .../net/ethernet/mellanox/mlx5/core/Kconfig | 1 + .../net/ethernet/mellanox/mlx5/core/Makefile | 1 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 21 +- .../mellanox/mlx5/core/en/netgpu/setup.c | 340 +++++ .../mellanox/mlx5/core/en/netgpu/setup.h | 96 ++ .../ethernet/mellanox/mlx5/core/en/params.c | 3 +- .../ethernet/mellanox/mlx5/core/en/params.h | 9 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 + .../ethernet/mellanox/mlx5/core/en/xsk/umem.c | 4 + .../ethernet/mellanox/mlx5/core/en/xsk/umem.h | 3 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 121 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 58 +- .../net/ethernet/mellanox/mlx5/core/en_tx.c | 19 + .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 16 +- include/linux/dma-mapping.h | 4 +- include/linux/log2.h | 2 +- include/linux/memory_hotplug.h | 4 + include/linux/mmzone.h | 7 + include/linux/netdevice.h | 6 + include/linux/skbuff.h | 27 +- include/linux/socket.h | 1 + include/linux/uio.h | 4 + include/net/netgpu.h | 66 + include/uapi/misc/netgpu.h | 69 + include/uapi/misc/shqueue.h | 200 +++ kernel/user.c | 1 + lib/iov_iter.c | 53 + mm/memory_hotplug.c | 65 +- net/core/datagram.c | 9 +- net/core/skbuff.c | 50 +- net/ipv4/tcp.c | 13 + net/ipv4/tcp_output.c | 20 + 45 files changed, 3827 insertions(+), 49 deletions(-) create mode 100644 drivers/misc/netgpu/Kconfig create mode 100644 drivers/misc/netgpu/Makefile create mode 100644 drivers/misc/netgpu/netgpu_host.c create mode 100644 drivers/misc/netgpu/netgpu_main.c create mode 100644 drivers/misc/netgpu/netgpu_mem.c create mode 100644 drivers/misc/netgpu/netgpu_priv.h create mode 100644 drivers/misc/netgpu/netgpu_stub.c create mode 100644 drivers/misc/netgpu/netgpu_stub.h create mode 100644 drivers/misc/netgpu/nvidia/Kbuild create mode 100644 drivers/misc/netgpu/nvidia/Kconfig create mode 100644 drivers/misc/netgpu/nvidia/netgpu_cuda.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.h create mode 100644 include/net/netgpu.h create mode 100644 include/uapi/misc/netgpu.h create mode 100644 include/uapi/misc/shqueue.h
Signature · 1 line
2.24.1
RFC PATCH v2 9/21 core/skbuff: use skb_zdata for testing whether skb is zerocopy
From: Jonathan Lemon <[email protected]>
skb_zcopy() flag indicates whether the skb has a zerocopy ubuf. netgpu does not use ubufs, so skb_zdata() indicates whether the skb is carrying zero copy data, and should be left alone, while skb_zcopy() indicates whether there is an attached ubuf.
--- include/linux/skbuff.h | 24 +++++++++++++++++++++++- net/core/skbuff.c | 17 ++++++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-)
include/linux/skbuff.h +23 -1
@@ -443,8 +443,12 @@ enum {/* generate software time stamp when entering packet scheduling */ SKBTX_SCHED_TSTAMP =1 <<6 , + +/* fragments are accessed only via DMA */ + SKBTX_DEV_NETDMA = 1 << 7, }; +#define SKBTX_ZERODATA_FRAG (SKBTX_DEV_ZEROCOPY | SKBTX_DEV_NETDMA)#define SKBTX_ZEROCOPY_FRAG (SKBTX_DEV_ZEROCOPY | SKBTX_SHARED_FRAG)#define SKBTX_ANY_SW_TSTAMP (SKBTX_SW_TSTAMP | \ SKBTX_SCHED_TSTAMP)
@@ -1420,6 +1424,24 @@ static inline struct skb_shared_hwtstamps *skb_hwtstamps(struct sk_buff *skb)return &skb_shinfo (skb)->hwtstamps; } +static inline bool skb_netdma (struct sk_buff *skb) +{ +return skb &&skb_shinfo (skb)->tx_flags &SKBTX_DEV_NETDMA ; +} + +static inline bool skb_zdata (struct sk_buff *skb) +{ +return skb &&skb_shinfo (skb)->tx_flags &SKBTX_ZERODATA_FRAG ; +} + +static inline void skb_netdma_set (struct sk_buff *skb,void *arg) +{ +if (skb && arg) { +skb_shinfo (skb)->tx_flags |=SKBTX_DEV_NETDMA ; +skb_shinfo (skb)->destructor_arg = arg; + } +} +static inline struct ubuf_info *skb_zcopy (struct sk_buff *skb) {bool is_zcopy = skb &&skb_shinfo (skb)->tx_flags &SKBTX_DEV_ZEROCOPY ;
@@ -3264,7 +3286,7 @@ static inline int skb_add_data(struct sk_buff *skb,static inline bool skb_can_coalesce (struct sk_buff *skb,int i,const struct page *page,int off) { -if (skb_zcopy (skb)) +if (skb_zdata (skb))return false;if (i) {const skb_frag_t *frag = &skb_shinfo (skb)->frags[i -1 ];
net/core/skbuff.c +16 -1
@@ -69,6 +69,7 @@#include <net/xfrm.h> #include <net/mpls.h> #include <net/mptcp.h> +#include <net/netgpu.h> #include <linux/uaccess.h> #include <trace/events/skb.h>
@@ -1300,6 +1301,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, }skb_zcopy_set (skb, uarg,NULL ); +skb_netdma_set (skb, sk->sk_user_data); +return skb->len - orig_len; }EXPORT_SYMBOL_GPL (skb_zerocopy_iter_stream);
@@ -1307,6 +1310,16 @@ EXPORT_SYMBOL_GPL(skb_zerocopy_iter_stream);static int skb_zerocopy_clone (struct sk_buff *nskb,struct sk_buff *orig,gfp_t gfp_mask) { +if (skb_netdma (orig)) { +if (skb_netdma (nskb)) { +WARN_ONCE (1 ,"zc clone, dst skb is set\n" ); +if (skb_uarg (nskb) !=skb_uarg (orig)) +return -EIO ; + } +skb_netdma_set (nskb,skb_shinfo (orig)->destructor_arg); +return 0 ; + } +if (skb_zcopy (orig)) {if (skb_zcopy (nskb)) {/* !gfp_mask callers are verified to !skb_zcopy(nskb) */
@@ -2055,6 +2068,8 @@ void *__pskb_pull_tail(struct sk_buff *skb, int delta)*/ int i, k, eat = (skb->tail + delta) - skb->end; +BUG_ON (skb_netdma (skb)); +if (eat >0 ||skb_cloned (skb)) {if (pskb_expand_head (skb,0 , eat >0 ? eat +128 :0 ,GFP_ATOMIC ))
@@ -3305,7 +3320,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)if (skb_headlen (skb))return 0 ; -if (skb_zcopy (tgt) ||skb_zcopy (skb)) +if (skb_zdata (tgt) ||skb_zdata (skb))return 0 ; todo = shiftlen;
Signature · 1 line
2.24.1
RFC PATCH v2 4/21 kernel/user: export free_uid
In reply to Jonathan Lemonget_uid is a static inline which can be called from a module, so free_uid should also be callable from a module.
--- kernel/user.c | 1 + 1 file changed, 1 insertion(+)
kernel/user.c +1 -0
@@ -171,6 +171,7 @@ void free_uid(struct user_struct *up)if (refcount_dec_and_lock_irqsave (&up->__count, &uidhash_lock, &flags))free_user (up, flags); } +EXPORT_SYMBOL_GPL (free_uid);struct user_struct *alloc_uid (kuid_t uid) {
Signature · 1 line
2.24.1
RFC PATCH v2 7/21 netdevice: add SETUP_NETGPU to the netdev_bpf structure
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
This command will be used to setup/tear down netgpu queues.
--- include/linux/netdevice.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/netdevice.h +6 -0
@@ -882,6 +882,7 @@ enum bpf_netdev_command {BPF_OFFLOAD_MAP_ALLOC ,BPF_OFFLOAD_MAP_FREE ,XDP_SETUP_XSK_UMEM , + XDP_SETUP_NETGPU, };struct bpf_prog_offload_ops ;
@@ -913,6 +914,11 @@ struct netdev_bpf {struct xdp_umem *umem;u16 queue_id; } xsk; +/* XDP_SETUP_NETGPU */ +struct { +struct netgpu_ifq *ifq; +u16 queue_id; + } netgpu; }; };
Signature · 1 line
2.24.1
RFC PATCH v2 10/21 netgpu: add network/gpu/host dma module
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Netgpu provides a data path for zero-copy sends and receives without having the host CPU touch the data. Protocol processing is done on the host CPU, while data is DMA'd to and from DMA mapped memory areas. The initial code provides transfers between (mlx5 / host memory) and (mlx5 / nvidia GPU memory).
The use case for this module are GPUs used for machine learning, which are located near the NICs, and have a high bandwidth PCI connection between the GPU/NIC.
--- drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/netgpu/Kconfig | 14 + drivers/misc/netgpu/Makefile | 6 + drivers/misc/netgpu/netgpu_host.c | 284 +++++++ drivers/misc/netgpu/netgpu_main.c | 1215 +++++++++++++++++++++++++++++ drivers/misc/netgpu/netgpu_mem.c | 351 +++++++++ drivers/misc/netgpu/netgpu_priv.h | 88 +++ drivers/misc/netgpu/netgpu_stub.c | 166 ++++ drivers/misc/netgpu/netgpu_stub.h | 19 + 10 files changed, 2145 insertions(+) create mode 100644 drivers/misc/netgpu/Kconfig create mode 100644 drivers/misc/netgpu/Makefile create mode 100644 drivers/misc/netgpu/netgpu_host.c create mode 100644 drivers/misc/netgpu/netgpu_main.c create mode 100644 drivers/misc/netgpu/netgpu_mem.c create mode 100644 drivers/misc/netgpu/netgpu_priv.h create mode 100644 drivers/misc/netgpu/netgpu_stub.c create mode 100644 drivers/misc/netgpu/netgpu_stub.h
drivers/misc/Kconfig +1 -0
@@ -472,4 +472,5 @@ source "drivers/misc/ocxl/Kconfig" source "drivers/misc/cardreader/Kconfig" source "drivers/misc/habanalabs/Kconfig" source "drivers/misc/uacce/Kconfig" +source "drivers/misc/netgpu/Kconfig" endmenu
drivers/misc/Makefile +1 -0
@@ -57,3 +57,4 @@ obj-$(CONFIG_PVPANIC) += pvpanic.o obj-$(CONFIG_HABANA_AI) += habanalabs/ obj-$(CONFIG_UACCE) += uacce/ obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o +obj-y += netgpu/
drivers/misc/netgpu/Kconfig new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# NetGPU framework +# +config NETGPU + tristate "Network/GPU driver" + depends on PCI + imply NETGPU_STUB + help + Experimental Network / GPU driver + +config NETGPU_STUB + bool + depends on NETGPU = m
drivers/misc/netgpu/Makefile new file +6 -0
@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0-only + +obj-$(CONFIG_NETGPU) := netgpu.o +netgpu-y := netgpu_mem.o netgpu_main.o netgpu_host.o + +obj-$(CONFIG_NETGPU_STUB) := netgpu_stub.o
drivers/misc/netgpu/netgpu_host.c new file +284 -0
284 lines not on this page · raw message
drivers/misc/netgpu/netgpu_main.c new file +1215 -0
1,215 lines not on this page · raw message
drivers/misc/netgpu/netgpu_mem.c new file +351 -0
351 lines not on this page · raw message
drivers/misc/netgpu/netgpu_priv.h new file +88 -0
88 lines not on this page · raw message
drivers/misc/netgpu/netgpu_stub.c new file +166 -0
166 lines not on this page · raw message
drivers/misc/netgpu/netgpu_stub.h new file +19 -0
@@ -0,0 +1,19 @@ +#pragma once + +/* development-only support for module loading. */ + +struct netgpu_functions { +dma_addr_t (*get_dma)(struct netgpu_ctx *ctx,struct page *page); +int (*get_page)(struct netgpu_ifq *ifq, +struct page **page,dma_addr_t *dma); +void (*put_page)(struct netgpu_ifq *,struct page *,bool ); +int (*get_pages)(struct sock *,struct page **, +unsigned long ,int ); + +int (*socket_mmap)(struct file *file,struct socket *sock, +struct vm_area_struct *vma); +int (*attach_socket)(struct sock *sk,void __user *arg); +}; + +void netgpu_fcn_register (struct netgpu_functions *f); +void netgpu_fcn_unregister (void );
Signature · 1 line
2.24.1
RFC PATCH v2 19/21 mlx5e: add the netgpu driver functions
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Add the netgpu queue setup/teardown functions, and the interface into the main netgpu core code. These will be hooked up to the mlx5 driver in the next commit.
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 1 + .../net/ethernet/mellanox/mlx5/core/Makefile | 1 + .../mellanox/mlx5/core/en/netgpu/setup.c | 340 ++++++++++++++++++ .../mellanox/mlx5/core/en/netgpu/setup.h | 96 +++++ .../ethernet/mellanox/mlx5/core/en/params.h | 8 + 5 files changed, 446 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +1 -0
@@ -33,6 +33,7 @@ config MLX5_FPGA config MLX5_CORE_EN bool "Mellanox 5th generation network adapters (ConnectX series) Ethernet support" depends on NETDEVICES && ETHERNET && INET && PCI && MLX5_CORE + depends on NETGPU || !NETGPU select PAGE_POOL select DIMLIB default n
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -0
@@ -41,6 +41,7 @@ mlx5_core-$(CONFIG_MLX5_CLS_ACT) += en_tc.o en/rep/tc.o en/rep/neigh.o \ en/tc_tun_vxlan.o en/tc_tun_gre.o en/tc_tun_geneve.o \ en/tc_tun_mplsoudp.o diag/en_tc_tracepoint.o mlx5_core-$(CONFIG_MLX5_TC_CT) += en/tc_ct.o +mlx5_core-$(CONFIG_NETGPU) += en/netgpu/setup.o # # Core extra
drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c new file +340 -0
340 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.h new file +96 -0
96 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +8 -0
@@ -67,6 +67,14 @@ static inline void mlx5e_qid_get_ch_and_group(struct mlx5e_params *params, *group = qid / nch; } +static inline void mlx5e_get_qid_for_ch_in_group (struct mlx5e_params *params, +u16 *qid, +u16 ix, +enum mlx5e_rq_group group) +{ + *qid = params->num_channels * group + ix; +} +static inline bool mlx5e_qid_validate (const struct mlx5e_profile *profile,struct mlx5e_params *params,u64 qid) {
Signature · 1 line
2.24.1
RFC PATCH v2 11/21 core/skbuff: add page recycling logic for netgpu pages
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
netgpu pages will always have a refcount of at least one (held by the netgpu module). If the skb is marked as containing netgpu ZC pages, recycle them back to netgpu.
--- net/core/skbuff.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
net/core/skbuff.c +30 -2
@@ -591,6 +591,27 @@ static void skb_free_head(struct sk_buff *skb)kfree (head); } +#if IS_ENABLED (CONFIG_NETGPU ) +static void skb_netgpu_unref (struct skb_shared_info *shinfo) +{ +struct netgpu_ifq *ifq = shinfo->destructor_arg; +struct page *page; +int i; + +/* pages attached for skbs for TX shouldn't come here, since +* the skb is not marked as "zc_netgpu". (only RX skbs have this). +* dummy page does come here, but always has elevated refc. +* +* Undelivered zc skb's will arrive at this point. +*/ +for (i =0 ; i < shinfo->nr_frags; i++) { + page =skb_frag_page (&shinfo->frags[i]); +if (page &&page_ref_dec_return (page) <=2 ) +netgpu_put_page (ifq, page, false); + } +} +#endif +static void skb_release_data (struct sk_buff *skb) {struct skb_shared_info *shinfo =skb_shinfo (skb);
@@ -601,8 +622,15 @@ static void skb_release_data(struct sk_buff *skb) &shinfo->dataref))return ; -for (i =0 ; i < shinfo->nr_frags; i++) -__skb_frag_unref (&shinfo->frags[i]); +#if IS_ENABLED (CONFIG_NETGPU ) +if (skb->zc_netgpu && shinfo->nr_frags) { +skb_netgpu_unref (shinfo); + } else +#endif + { +for (i =0 ; i < shinfo->nr_frags; i++) +__skb_frag_unref (&shinfo->frags[i]); + }if (shinfo->frag_list)kfree_skb_list (shinfo->frag_list);
Signature · 1 line
2.24.1
RFC PATCH v2 12/21 lib: have __zerocopy_sg_from_iter get netgpu pages for a sk
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
If a sock is marked as sending zc data, have the iterator retrieve the correct zc pages from the netgpu module.
--- include/linux/uio.h | 4 ++++ lib/iov_iter.c | 53 +++++++++++++++++++++++++++++++++++++++++++++ net/core/datagram.c | 9 ++++++-- 3 files changed, 64 insertions(+), 2 deletions(-)
include/linux/uio.h +4 -0
@@ -227,6 +227,10 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages,ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages); +struct sock ; +ssize_t iov_iter_sk_get_pages (struct iov_iter *i,struct page **pages, +size_t maxsize,unsigned maxpages,size_t *pgoff, +struct sock *sk);const void *dup_iter (struct iov_iter *new,struct iov_iter *old,gfp_t flags);
lib/iov_iter.c +53 -0
65 lines not on this page · raw message
net/core/datagram.c +7 -2
@@ -530,6 +530,10 @@ int skb_copy_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len) {trace_skb_copy_datagram_iovec (skb ,len ); +if (skb->zc_netgpu) { +pr_err ("skb netgpu datagram on !netgpu sk\n" ); +return -EFAULT ; + }return __skb_datagram_iter (skb, offset, to, len, false, simple_copy_to_iter,NULL ); }
@@ -631,8 +635,9 @@ int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb,if (frag ==MAX_SKB_FRAGS )return -EMSGSIZE ; - copied =iov_iter_get_pages (from, pages, length, -MAX_SKB_FRAGS - frag, &start); + copied =iov_iter_sk_get_pages (from, pages, length, +MAX_SKB_FRAGS - frag, &start, + sk);if (copied <0 )return -EFAULT ;
Signature · 1 line
2.24.1
RFC PATCH v2 13/21 net/tcp: Pad TCP options out to a fixed size for netgpu
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
The "header splitting" feature used by netgpu doesn't actually parse the incoming packet header. Instead, it splits the packet at a fixed offset. In order for this to work, the sender needs to send packets with a fixed header size.
--- net/ipv4/tcp_output.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
net/ipv4/tcp_output.c +20 -0
@@ -438,6 +438,7 @@ struct tcp_out_options {u8 ws;/* window scale, 0 to disable */ u8 num_sack_blocks;/* number of SACK blocks to include */ u8 hash_size;/* bytes in hash_location */ +u8 pad_size;/* additional nops for padding */ __u8 *hash_location;/* temporary pointer, overloaded */ __u32 tsval, tsecr;/* need to include OPTION_TS */ struct tcp_fastopen_cookie *fastopen_cookie;/* Fast open cookie */
@@ -562,6 +563,17 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,smc_options_write (ptr, &options);mptcp_options_write (ptr, opts); + +#if IS_ENABLED (CONFIG_NETGPU ) +/* pad out options */ +if (opts->pad_size) { +int len = opts->pad_size; +u8 *p = (u8 *)ptr; + +while (len--) + *p++ =TCPOPT_NOP ; + } +#endif } static void smc_set_option(const struct tcp_sock *tp,
@@ -826,6 +838,14 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb opts->num_sack_blocks *TCPOLEN_SACK_PERBLOCK ; } +#if IS_ENABLED (CONFIG_NETGPU ) +/* force padding */ +if (size <20 ) { + opts->pad_size =20 - size; + size += opts->pad_size; + } +#endif +return size; }
Signature · 1 line
2.24.1
RFC PATCH v2 8/21 skbuff: add a zc_netgpu bitflag
In reply to Jonathan LemonThis could likely be moved elsewhere. The presence of the flag on the skb indicates that one of the fragments may contain zerocopy RX data, where the data is not accessible to the cpu.
--- include/linux/skbuff.h | 3 ++- net/core/skbuff.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/skbuff.h +2 -1
@@ -782,7 +782,8 @@ struct sk_buff { fclone:2, peeked:1, head_frag:1, - pfmemalloc:1; + pfmemalloc:1, + zc_netgpu:1;#ifdef CONFIG_SKB_EXTENSIONS __u8 active_extensions;#endif
net/core/skbuff.c +1 -0
@@ -992,6 +992,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb) n->cloned =1 ; n->nohdr =0 ; n->peeked =0 ; +C (zc_netgpu);C (pfmemalloc); n->destructor =NULL ;C (tail);
Signature · 1 line
2.24.1
RFC PATCH v2 5/21 uapi/misc: add shqueue.h for shared queues
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Shared queues between user and kernel use their own private structures for accessing a shared data area, but they need to use the same queue functions.
Have the kernel use the same UAPI file - this can be made private at a later date if required.
--- include/uapi/misc/shqueue.h | 200 ++++++++++++++++++++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 include/uapi/misc/shqueue.h
include/uapi/misc/shqueue.h new file +200 -0
@@ -0,0 +1,200 @@ +#ifndef _UAPI_MISC_SHQUEUE_H +#define _UAPI_MISC_SHQUEUE_H + +/* Placed under UAPI in order to avoid two identical copies between +* user and kernel space. +*/ + +/* user and kernel private copy - identical in order to share sq* fcns */ +struct shared_queue { +unsigned *prod; +unsigned *cons; +unsigned char *data; +unsigned elt_sz; +unsigned mask; +unsigned cached_prod; +unsigned cached_cons; +unsigned entries; + +unsigned map_sz; +void *map_ptr; +}; + +/* +* see documenation in tools/include/linux/ring_buffer.h +* using explicit smp_/_ONCE is an optimization over smp_{store|load} +*/ + +static inline void __sq_load_acquire_cons (struct shared_queue *q) +{ +/* Refresh the local tail pointer */ + q->cached_cons =READ_ONCE (*q->cons); +/* A, matches D */ +} + +static inline void __sq_store_release_cons (struct shared_queue *q) +{ +smp_mb ();/* D, matches A */ +WRITE_ONCE (*q->cons, q->cached_cons); +} + +static inline void __sq_load_acquire_prod (struct shared_queue *q) +{ +/* Refresh the local pointer */ + q->cached_prod =READ_ONCE (*q->prod); +smp_rmb ();/* C, matches B */ +} + +static inline void __sq_store_release_prod (struct shared_queue *q) +{ +smp_wmb ();/* B, matches C */ +WRITE_ONCE (*q->prod, q->cached_prod); +} + +static inline void sq_cons_refresh (struct shared_queue *q) +{ +__sq_store_release_cons (q); +__sq_load_acquire_prod (q); +} + +static inline bool sq_is_empty (struct shared_queue *q) +{ +return READ_ONCE (*q->prod) ==READ_ONCE (*q->cons); +} + +static inline bool sq_cons_empty (struct shared_queue *q) +{ +return q->cached_prod == q->cached_cons; +} + +static inline unsigned __sq_cons_ready (struct shared_queue *q) +{ +return q->cached_prod - q->cached_cons; +} + +static inline unsigned sq_cons_ready (struct shared_queue *q) +{ +if (sq_cons_empty (q)) +__sq_load_acquire_prod (q); + +return __sq_cons_ready (q); +} + +static inline bool sq_cons_avail (struct shared_queue *q,unsigned count) +{ +if (count <=__sq_cons_ready (q)) +return true; +__sq_load_acquire_prod (q); +return count <=__sq_cons_ready (q); +} + +static inline void *sq_get_ptr (struct shared_queue *q,unsigned idx) +{ +return q->data + (idx & q->mask) * q->elt_sz; +} + +static inline void sq_cons_complete (struct shared_queue *q) +{ +__sq_store_release_cons (q); +} + +static inline void *sq_cons_peek (struct shared_queue *q) +{ +if (sq_cons_empty (q)) { +sq_cons_refresh (q); +if (sq_cons_empty (q)) +return NULL ; + } +return sq_get_ptr (q, q->cached_cons); +} + +static inline unsigned +sq_peek_batch (struct shared_queue *q,void **ptr,unsigned count) +{ +unsigned i, idx, ready; + + ready =sq_cons_ready (q); +if (!ready) +return 0 ; + + count = count > ready ? ready : count; + + idx = q->cached_cons; +for (i =0 ; i < count; i++) + ptr[i] =sq_get_ptr (q, idx++); + + q->cached_cons += count; + +return count; +} + +static inline unsigned +sq_cons_batch (struct shared_queue *q,void **ptr,unsigned count) +{ +unsigned i, idx, ready; + + ready =sq_cons_ready (q); +if (!ready) +return 0 ; + + count = count > ready ? ready : count; + + idx = q->cached_cons; +for (i =0 ; i < count; i++) + ptr[i] =sq_get_ptr (q, idx++); + + q->cached_cons += count; +sq_cons_complete (q); + +return count; +} + +static inline void sq_cons_advance (struct shared_queue *q) +{ + q->cached_cons++; +} + +static inline unsigned __sq_prod_space (struct shared_queue *q) +{ +return q->entries - (q->cached_prod - q->cached_cons); +} + +static inline unsigned sq_prod_space (struct shared_queue *q) +{ +unsigned space; + + space =__sq_prod_space (q); +if (!space) { +__sq_load_acquire_cons (q); + space =__sq_prod_space (q); + } +return space; +} + +static inline bool sq_prod_avail (struct shared_queue *q,unsigned count) +{ +if (count <=__sq_prod_space (q)) +return true; +__sq_load_acquire_cons (q); +return count <=__sq_prod_space (q); +} + +static inline void *sq_prod_get_ptr (struct shared_queue *q) +{ +return sq_get_ptr (q, q->cached_prod++); +} + +static inline void *sq_prod_reserve (struct shared_queue *q) +{ +if (!sq_prod_space (q)) +return NULL ; + +return sq_prod_get_ptr (q); +} + +static inline void sq_prod_submit (struct shared_queue *q) +{ +__sq_store_release_prod (q); +} + +#endif /* _UAPI_MISC_SHQUEUE_H */
Signature · 1 line
2.24.1
RFC PATCH v2 15/21 net/tcp: add MSG_NETDMA flag for sendmsg()
In reply to Jonathan LemonThis flag indicates that the attached data is a zero-copy send, and the pages should be retrieved from the netgpu module. The socket should should already have been attached to a netgpu queue.
--- include/linux/socket.h | 1 + net/ipv4/tcp.c | 8 ++++++++ 2 files changed, 9 insertions(+)
include/linux/socket.h +1 -0
@@ -310,6 +310,7 @@ struct ucred {*/ #define MSG_ZEROCOPY 0x4000000/* Use user data in kernel path */ +#define MSG_NETDMA 0x8000000#define MSG_FASTOPEN 0x20000000/* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000/* Set close_on_exec for file descriptor received through
net/ipv4/tcp.c +8 -0
@@ -1214,6 +1214,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) uarg->zerocopy =0 ; } +if (flags &MSG_NETDMA && size &&sock_flag (sk,SOCK_ZEROCOPY )) { + zc = sk->sk_route_caps &NETIF_F_SG ; +if (!zc) { + err = -EFAULT ; + goto out_err; + } + } +if (unlikely (flags &MSG_FASTOPEN ||inet_sk (sk)->defer_connect) && !tp->repair) { err =tcp_sendmsg_fastopen (sk, msg, &copied_syn, size, uarg);
Signature · 1 line
2.24.1
RFC PATCH v2 16/21 mlx5: remove the umem parameter from mlx5e_open_channel
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Instead of obtaining the umem parameter from the channel parameters and passing it to the function, push this down into the function itself.
Move xsk open logic into its own function, in preparation for the upcoming netgpu commit.
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 37 +++++++++++++------ 1 file changed, 25 insertions(+), 12 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +25 -12
@@ -393,7 +393,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->xdpsq = &c->rq_xdpsq; rq->umem = umem; -if (rq->umem) +if (xsk) rq->stats = &c->priv->channel_stats[c->ix].xskrq;else rq->stats = &c->priv->channel_stats[c->ix].rq;
@@ -1946,15 +1946,33 @@ static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)return (ix + port_aff_bias) %mlx5e_get_num_lag_ports (mdev); } +static int +mlx5e_xsk_optional_open (struct mlx5e_priv *priv,int ix, +struct mlx5e_params *params, +struct mlx5e_channel_param *cparam, +struct mlx5e_channel *c) +{ +struct mlx5e_xsk_param xsk; +struct xdp_umem *umem; +int err =0 ; + + umem =mlx5e_xsk_get_umem (params, params->xsk, ix); + +if (umem) { +mlx5e_build_xsk_param (umem, &xsk); + err =mlx5e_open_xsk (priv, params, &xsk, umem, c); + } + +return err; +} +static int mlx5e_open_channel (struct mlx5e_priv *priv,int ix,struct mlx5e_params *params,struct mlx5e_channel_param *cparam, -struct xdp_umem *umem,struct mlx5e_channel **cp) {int cpu =cpumask_first (mlx5_comp_irq_get_affinity_mask (priv->mdev, ix));struct net_device *netdev = priv->netdev; -struct mlx5e_xsk_param xsk;struct mlx5e_channel *c;unsigned int irq;int err;
@@ -1988,9 +2006,9 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,if (unlikely (err)) goto err_napi_del; -if (umem) { -mlx5e_build_xsk_param (umem, &xsk); - err =mlx5e_open_xsk (priv, params, &xsk, umem, c); +/* This opens a second set of shadow queues for xsk */ +if (params->xdp_prog) { + err =mlx5e_xsk_optional_open (priv, ix, params, cparam, c);if (unlikely (err)) goto err_close_queues; }
@@ -2351,12 +2369,7 @@ int mlx5e_open_channels(struct mlx5e_priv *priv,mlx5e_build_channel_param (priv, &chs->params, cparam);for (i =0 ; i < chs->num; i++) { -struct xdp_umem *umem =NULL ; - -if (chs->params.xdp_prog) - umem =mlx5e_xsk_get_umem (&chs->params, chs->params.xsk, i); - - err =mlx5e_open_channel (priv, i, &chs->params, cparam, umem, &chs->c[i]); + err =mlx5e_open_channel (priv, i, &chs->params, cparam, &chs->c[i]);if (err) goto err_close_channels; }
Signature · 1 line
2.24.1
RFC PATCH v2 18/21 mlx5e: add netgpu entries to mlx5 structures
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Modify mlx5e structures in order to add support for netgpu, which shares some of the same structures as AF_XDP. Add logic to make sure they are not both in use.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 12 ++++++++++-- .../net/ethernet/mellanox/mlx5/core/en/xsk/umem.c | 3 +++ .../net/ethernet/mellanox/mlx5/core/en/xsk/umem.h | 3 +++ drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 ++- 4 files changed, 18 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +10 -2
@@ -365,6 +365,7 @@ struct mlx5e_dma_info {struct page *page;struct xdp_buff *xsk; }; +bool netgpu_source; };/* XDP packets can be transmitted in different ways. On completion, we need to
@@ -553,6 +554,7 @@ struct mlx5e_rq_frags_info {u8 wqe_bulk; }; +struct netgpu_ifq ; struct mlx5e_rq { /* data path */ union {
@@ -608,8 +610,9 @@ struct mlx5e_rq {DECLARE_BITMAP (flags,8 );struct page_pool *page_pool; -/* AF_XDP zero-copy */ +/* AF_XDP or NETGPU zero-copy */ struct xdp_umem *umem; +struct netgpu_ifq *netgpu;struct work_struct recover_work;
@@ -627,6 +630,7 @@ struct mlx5e_rq {enum mlx5e_channel_state {MLX5E_CHANNEL_STATE_XSK , +MLX5E_CHANNEL_STATE_NETGPU ,MLX5E_CHANNEL_NUM_STATES };
@@ -737,9 +741,13 @@ struct mlx5e_xsk {* but it doesn't distinguish between zero-copy and non-zero-copy UMEMs, * so rely on our mechanism. */ -struct xdp_umem **umems; +union { +struct xdp_umem **umems; +struct netgpu_ifq **ifq_tbl; + };u16 refcnt;bool ever_used; +bool is_umem; };/* Temporary storage for variables that are allocated when struct mlx5e_priv is
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c +3 -0
@@ -27,7 +27,10 @@ static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk) sizeof(*xsk->umems), GFP_KERNEL);if (unlikely (!xsk->umems))return -ENOMEM ; + xsk->is_umem = true; } +if (!xsk->is_umem) +return -EINVAL ; xsk->refcnt++; xsk->ever_used = true;
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h +3 -0
@@ -15,6 +15,9 @@ static inline struct xdp_umem *mlx5e_xsk_get_umem(struct mlx5e_params *params,if (unlikely (ix >= params->num_channels))return NULL ; +if (unlikely (!xsk->is_umem)) +return NULL ; +return xsk->umems[ix]; }
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -1
@@ -62,7 +62,6 @@#include "en/xsk/setup.h" #include "en/xsk/rx.h" #include "en/xsk/tx.h" -#include "en/netgpu/setup.h" #include "en/hv_vhca_stats.h" #include "en/devlink.h" #include "lib/mlx5.h"
@@ -324,6 +323,8 @@ static void mlx5e_init_frags_partition(struct mlx5e_rq *rq)if (prev) prev->last_in_page = true; } + next_frag.di->netgpu_source = + !!frag_info[f].frag_source; *frag = next_frag;/* prepare next */
Signature · 1 line
2.24.1
RFC PATCH v2 3/21 mm: Allow DMA mapping of pages which are not online
In reply to Jonathan LemonChange the system RAM check from 'valid' to 'online', so dummy pages which refer to external DMA resources can be mapped.
--- include/linux/dma-mapping.h | 4 ++-- include/linux/mmzone.h | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
include/linux/dma-mapping.h +2 -2
@@ -348,8 +348,8 @@ static inline dma_addr_t dma_map_resource(struct device *dev,BUG_ON (!valid_dma_direction (dir)); -/* Don't allow RAM to be mapped */ -if (WARN_ON_ONCE (pfn_valid (PHYS_PFN (phys_addr)))) +/* Don't allow online RAM to be mapped */ +if (WARN_ON_ONCE (pfn_online (PHYS_PFN (phys_addr))))return DMA_MAPPING_ERROR ;if (dma_is_direct (ops))
include/linux/mmzone.h +7 -0
@@ -1348,6 +1348,13 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr)return -1 ; } +static inline int pfn_online (unsigned long pfn) +{ +if (pfn_to_section_nr (pfn) >=NR_MEM_SECTIONS ) +return 0 ; +return online_section (__nr_to_section (pfn_to_section_nr (pfn))); +} +/* * These are _only_ used during initialisation, therefore they * can use __initdata ... They could have names to indicate
Signature · 1 line
2.24.1
RFC PATCH v2 1/21 linux/log2.h: enclose macro arg in parens
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
roundup_pow_of_two uses its arg without enclosing it in parens.
A call of the form:
roundup_pow_of_two(boolval ? PAGE_SIZE : frag_size)
resulted in an compile warning:
warning: ?: using integer constants in boolean context [-Wint-in-bool-context]
PAGE_SIZE :../include/linux/log2.h:176:4: note: in definition of macro ‘roundup_pow_of_two’
(n == 1) ? 1 : \
^And the resulting code used '1' as the result of the operation.
--- include/linux/log2.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/log2.h +1 -1
@@ -173,7 +173,7 @@ unsigned long __rounddown_pow_of_two(unsigned long n)#define roundup_pow_of_two (n) \ ( \ __builtin_constant_p(n) ? ( \ - (n == 1) ? 1 : \ + ((n) == 1) ? 1 : \ (1UL << (ilog2((n) - 1) + 1)) \ ) : \ __roundup_pow_of_two(n) \
Signature · 1 line
2.24.1
RFC PATCH v2 6/21 include: add netgpu UAPI and kernel definitions
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
This provides the interface to the netgpu module.
--- include/net/netgpu.h | 66 ++++++++++++++++++++++++++++++++++++ include/uapi/misc/netgpu.h | 69 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) create mode 100644 include/net/netgpu.h create mode 100644 include/uapi/misc/netgpu.h
include/net/netgpu.h new file +66 -0
@@ -0,0 +1,66 @@ +#ifndef _NET_NETGPU_H +#define _NET_NETGPU_H + +#include <uapi/misc/netgpu.h> /* IOCTL defines */ +#include <uapi/misc/shqueue.h> + +enum { +NETGPU_MEMTYPE_HOST , +NETGPU_MEMTYPE_CUDA , + +NETGPU_MEMTYPE_MAX , +}; + +struct netgpu_pgcache { +struct netgpu_pgcache *next; +struct page *page[]; +}; + +struct netgpu_ifq { +struct shared_queue fill; +struct wait_queue_head fill_wait; +struct netgpu_ctx *ctx; +int queue_id; +spinlock_t pgcache_lock; +struct netgpu_pgcache *napi_cache; +struct netgpu_pgcache *spare_cache; +struct netgpu_pgcache *any_cache; +int napi_cache_count; +int any_cache_count; +struct list_head ifq_node; +}; + +struct netgpu_skq { +struct shared_queue rx; +struct shared_queue cq;/* for requested completions */ +struct netgpu_ctx *ctx; +void (*sk_destruct)(struct sock *sk); +void (*sk_data_ready)(struct sock *sk); +}; + +struct netgpu_ctx { +struct xarray xa;/* contains dmamaps */ +refcount_t ref; +struct net_device *dev; +struct list_head ifq_list; +}; + +struct net_device ; +struct netgpu_ops ; +struct socket ; + +dma_addr_t netgpu_get_dma (struct netgpu_ctx *ctx,struct page *page); +int netgpu_get_page (struct netgpu_ifq *ifq,struct page **page, +dma_addr_t *dma); +void netgpu_put_page (struct netgpu_ifq *ifq,struct page *page,bool napi); +int netgpu_get_pages (struct sock *sk,struct page **pages,unsigned long addr, +int count); + +int netgpu_socket_mmap (struct file *file,struct socket *sock, +struct vm_area_struct *vma); +int netgpu_attach_socket (struct sock *sk,void __user *arg); + +int netgpu_register (struct netgpu_ops *ops); +void netgpu_unregister (int memtype); + +#endif /* _NET_NETGPU_H */
include/uapi/misc/netgpu.h new file +69 -0
69 lines not on this page · raw message
Signature · 1 line
2.24.1
RFC PATCH v2 17/21 mlx5e: add header split ability
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Header split may be requested for a specific rq via a flag in the xsk parameter. If splitting is enabled (defaults to ipv6), set the wq_type to WQ_TYPE_CYCLIC.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 6 +++ .../ethernet/mellanox/mlx5/core/en/params.c | 3 +- .../ethernet/mellanox/mlx5/core/en/params.h | 1 + .../ethernet/mellanox/mlx5/core/en/xsk/umem.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 47 ++++++++++++++----- 5 files changed, 45 insertions(+), 13 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +6 -0
@@ -58,6 +58,11 @@extern const struct net_device_ops mlx5e_netdev_ops;struct page_pool ; +#define TCP_HDRS_LEN (20 + 20)/* headers + options */ +#define IP6_HDRS_LEN (40) +#define MAC_HDR_LEN (14) +#define TOTAL_HEADERS (TCP_HDRS_LEN + IP6_HDRS_LEN + MAC_HDR_LEN) +#define HD_SPLIT_DEFAULT_FRAG_SIZE (4096)#define MLX5E_METADATA_ETHER_TYPE (0x8CE4)#define MLX5E_METADATA_ETHER_LEN 8
@@ -538,6 +543,7 @@ enum mlx5e_rq_flag {struct mlx5e_rq_frag_info {int frag_size;int frag_stride; +int frag_source; };struct mlx5e_rq_frags_info {
drivers/net/ethernet/mellanox/mlx5/core/en/params.c +2 -1
@@ -146,7 +146,8 @@ u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev,struct mlx5e_params *params, struct mlx5e_xsk_param *xsk) { - bool is_linear_skb = (params->rq_wq_type == MLX5_WQ_TYPE_CYCLIC) ? + bool is_linear_skb = (xsk && xsk->hd_split) ? false : + (params->rq_wq_type ==MLX5_WQ_TYPE_CYCLIC ) ?mlx5e_rx_is_linear_skb (params, xsk) :mlx5e_rx_mpwqe_is_linear_skb (mdev, params, xsk);
drivers/net/ethernet/mellanox/mlx5/core/en/params.h +1 -0
@@ -9,6 +9,7 @@struct mlx5e_xsk_param {u16 headroom;u16 chunk_size; +bool hd_split; };struct mlx5e_cq_param {
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c +1 -0
@@ -72,6 +72,7 @@ void mlx5e_build_xsk_param(struct xdp_umem *umem, struct mlx5e_xsk_param *xsk) { xsk->headroom =xsk_umem_get_headroom (umem); xsk->chunk_size =xsk_umem_get_chunk_size (umem); + xsk->hd_split = false; } static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv,
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +35 -12
127 lines not on this page · raw message
Signature · 1 line
2.24.1
RFC PATCH v2 20/21 mlx5e: hook up the netgpu functions
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Hook up all the netgpu functions to the mlx5e driver.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 3 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 36 ++++++++++++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 58 ++++++++++++++++--- .../net/ethernet/mellanox/mlx5/core/en_tx.c | 19 ++++++ .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 16 ++++- 6 files changed, 125 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +2 -1
@@ -297,7 +297,8 @@ struct mlx5e_cq_decomp {enum mlx5e_dma_map_type {MLX5E_DMA_MAP_SINGLE , -MLX5E_DMA_MAP_PAGE +MLX5E_DMA_MAP_PAGE , +MLX5E_DMA_MAP_FIXED };struct mlx5e_sq_dma {
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +3 -0
@@ -253,6 +253,9 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)case MLX5E_DMA_MAP_PAGE :dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ; +case MLX5E_DMA_MAP_FIXED : +/* DMA mappings are fixed, or managed elsewhere. */ +break ;default :WARN_ONCE (true,"mlx5e_tx_dma_unmap unknown DMA type!\n" ); }
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +36 -0
@@ -62,6 +62,7 @@#include "en/xsk/setup.h" #include "en/xsk/rx.h" #include "en/xsk/tx.h" +#include "en/netgpu/setup.h" #include "en/hv_vhca_stats.h" #include "en/devlink.h" #include "lib/mlx5.h"
@@ -1970,6 +1971,24 @@ mlx5e_xsk_optional_open(struct mlx5e_priv *priv, int ix,return err; } +static int +mlx5e_netgpu_optional_open (struct mlx5e_priv *priv,int ix, +struct mlx5e_params *params, +struct mlx5e_channel_param *cparam, +struct mlx5e_channel *c) +{ +struct netgpu_ifq *ifq; +int err =0 ; + + ifq =mlx5e_netgpu_get_ifq (params, params->xsk, ix); + +if (ifq) + err =mlx5e_open_netgpu (priv, params, ifq, c); + +return err; +} + + static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, struct mlx5e_params *params, struct mlx5e_channel_param *cparam,
@@ -2017,6 +2036,11 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, goto err_close_queues; } +/* This opens a second set of shadow queues for netgpu */ + err =mlx5e_netgpu_optional_open (priv, ix, params, cparam, c); +if (unlikely (err)) + goto err_close_queues; + *cp = c;return 0 ;
@@ -2053,6 +2077,9 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)if (test_bit (MLX5E_CHANNEL_STATE_XSK , c->state))mlx5e_deactivate_xsk (c); +if (test_bit (MLX5E_CHANNEL_STATE_NETGPU , c->state)) +mlx5e_deactivate_netgpu (c); +mlx5e_deactivate_rq (&c->rq);mlx5e_deactivate_icosq (&c->async_icosq);mlx5e_deactivate_icosq (&c->icosq);
@@ -2064,6 +2091,10 @@ static void mlx5e_close_channel(struct mlx5e_channel *c) {if (test_bit (MLX5E_CHANNEL_STATE_XSK , c->state))mlx5e_close_xsk (c); + +if (test_bit (MLX5E_CHANNEL_STATE_NETGPU , c->state)) +mlx5e_close_netgpu (c); +mlx5e_close_queues (c);netif_napi_del (&c->napi);
@@ -3042,11 +3073,13 @@ void mlx5e_activate_priv_channels(struct mlx5e_priv *priv)mlx5e_redirect_rqts_to_channels (priv, &priv->channels);mlx5e_xsk_redirect_rqts_to_channels (priv, &priv->channels); +mlx5e_netgpu_redirect_rqts_to_channels (priv, &priv->channels); }void mlx5e_deactivate_priv_channels (struct mlx5e_priv *priv) {mlx5e_xsk_redirect_rqts_to_drop (priv, &priv->channels); +mlx5e_netgpu_redirect_rqts_to_drop (priv, &priv->channels);mlx5e_redirect_rqts_to_drop (priv);
@@ -4581,6 +4614,9 @@ static int mlx5e_xdp(struct net_device *dev, struct netdev_bpf *xdp)case XDP_SETUP_XSK_UMEM :return mlx5e_xsk_setup_umem (dev, xdp->xsk.umem, xdp->xsk.queue_id); +case XDP_SETUP_NETGPU : +return mlx5e_netgpu_setup_ifq (dev, xdp->netgpu.ifq, + &xdp->netgpu.queue_id);default :return -EINVAL ; }
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +50 -8
116 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +19 -0
@@ -39,6 +39,7 @@#include "ipoib/ipoib.h" #include "en_accel/en_accel.h" #include "lib/clock.h" +#include "en/netgpu/setup.h" static void mlx5e_dma_unmap_wqe_err (struct mlx5e_txqsq *sq,u8 num_dma) {
@@ -207,6 +208,24 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb, dseg++; } +if (skb_netdma (skb)) { +for (i =0 ; i <skb_shinfo (skb)->nr_frags; i++) { +skb_frag_t *frag = &skb_shinfo (skb)->frags[i]; +int fsz =skb_frag_size (frag); + + dma_addr =mlx5e_netgpu_get_dma (skb, frag); + + dseg->addr =cpu_to_be64 (dma_addr); + dseg->lkey = sq->mkey_be; + dseg->byte_count =cpu_to_be32 (fsz); + +mlx5e_dma_push (sq, dma_addr, fsz,MLX5E_DMA_MAP_FIXED ); + num_dma++; + dseg++; + } +return num_dma; + } +for (i =0 ; i <skb_shinfo (skb)->nr_frags; i++) {skb_frag_t *frag = &skb_shinfo (skb)->frags[i];int fsz =skb_frag_size (frag);
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +15 -1
@@ -122,6 +122,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)struct mlx5e_rq *xskrq = &c->xskrq;struct mlx5e_rq *rq = &c->rq;bool xsk_open =test_bit (MLX5E_CHANNEL_STATE_XSK , c->state); +bool netgpu_open =test_bit (MLX5E_CHANNEL_STATE_NETGPU , c->state);bool aff_change = false;bool busy_xsk = false;bool busy = false;
@@ -139,7 +140,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) busy |=mlx5e_poll_xdpsq_cq (&c->rq_xdpsq.cq);if (likely (budget)) {/* budget=0 means: don't poll rx rings */ -if (xsk_open) +if (xsk_open || netgpu_open) work_done =mlx5e_poll_rx_cq (&xskrq->cq, budget);if (likely (budget - work_done))
@@ -159,6 +160,14 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) mlx5e_post_rx_mpwqes, mlx5e_post_rx_wqes, rq); + +if (netgpu_open) { + busy_xsk |=INDIRECT_CALL_2 (xskrq->post_wqes, + mlx5e_post_rx_mpwqes, + mlx5e_post_rx_wqes, + xskrq); + } +if (xsk_open) { busy |=mlx5e_poll_xdpsq_cq (&xsksq->cq); busy_xsk |=mlx5e_napi_xsk_post (xsksq, xskrq);
@@ -192,6 +201,11 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)mlx5e_cq_arm (&c->async_icosq.cq);mlx5e_cq_arm (&c->xdpsq.cq); +if (netgpu_open) { +mlx5e_handle_rx_dim (xskrq); +mlx5e_cq_arm (&xskrq->cq); + } +if (xsk_open) {mlx5e_handle_rx_dim (xskrq);mlx5e_cq_arm (&xsksq->cq);
Signature · 1 line
2.24.1
RFC PATCH v2 14/21 net/tcp: add netgpu ioctl setting up zero copy RX queues
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
Netgpu delivers iovecs to userspace for incoming data, but the destination queue must be attached to the socket. Do this via and ioctl call on the socket itself.
--- net/ipv4/tcp.c | 5 +++++ 1 file changed, 5 insertions(+)
net/ipv4/tcp.c +5 -0
@@ -279,6 +279,7 @@#include <linux/uaccess.h> #include <asm/ioctls.h> #include <net/busy_poll.h> +#include <net/netgpu.h> struct percpu_counter tcp_orphan_count;EXPORT_SYMBOL_GPL (tcp_orphan_count);
@@ -636,6 +637,10 @@ int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) answ =READ_ONCE (tp->write_seq) -READ_ONCE (tp->snd_nxt);break ; +#if IS_ENABLED (CONFIG_NETGPU ) +case NETGPU_SOCK_IOCTL_ATTACH_QUEUES :/* SIOCPROTOPRIVATE */ +return netgpu_attach_socket (sk, (void __user *)arg); +#endif default :return -ENOIOCTLCMD ; }
Signature · 1 line
2.24.1
RFC PATCH v2 21/21 netgpu/nvidia: add Nvidia plugin for netgpu
In reply to Jonathan LemonFrom: Jonathan Lemon <[email protected]>
This provides the interface between the netgpu core module and the nvidia kernel driver. This should be built as an external module, pointing to the nvidia build. For example:
export NV_PACKAGE_DIR=/w/nvidia/NVIDIA-Linux-x86_64-440.64 make -C ${kdir} M=pwd O=obj $*
--- drivers/misc/netgpu/nvidia/Kbuild | 9 + drivers/misc/netgpu/nvidia/Kconfig | 10 + drivers/misc/netgpu/nvidia/netgpu_cuda.c | 416 +++++++++++++++++++++++ 3 files changed, 435 insertions(+) create mode 100644 drivers/misc/netgpu/nvidia/Kbuild create mode 100644 drivers/misc/netgpu/nvidia/Kconfig create mode 100644 drivers/misc/netgpu/nvidia/netgpu_cuda.c
drivers/misc/netgpu/nvidia/Kbuild new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-only + +nv_dir = $(NV_PACKAGE_DIR)/kernel + +KBUILD_EXTRA_SYMBOLS = $(nv_dir)/Module.symvers + +obj-m := netgpu_cuda.o + +ccflags-y += -I$(nv_dir)
drivers/misc/netgpu/nvidia/Kconfig new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# NetGPU framework +# + +config NETGPU_CUDA + tristate "Network/GPU driver for Nvidia" + depends on NETGPU && m + help + Experimental Network / GPU driver for Nvidia
drivers/misc/netgpu/nvidia/netgpu_cuda.c new file +416 -0
416 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [RFC PATCH v2 14/21] net/tcp: add netgpu ioctl setting up zero copy RX queues
In reply to Jonathan LemonQuoted 24 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
Netgpu delivers iovecs to userspace for incoming data, but the destination queue must be attached to the socket. Do this via and ioctl call on the socket itself.
Signed-off-by Jonathan Lemon <[email protected]>--- net/ipv4/tcp.c | 5 +++++ 1 file changed, 5 insertions(+)net/ipv4/tcp.c +1
-0@@ -279,6 +279,7 @@#include <linux/uaccess.h> #include <asm/ioctls.h> #include <net/busy_poll.h> +#include <net/netgpu.h> struct percpu_counter tcp_orphan_count;EXPORT_SYMBOL_GPL (tcp_orphan_count);
net/ipv4/tcp.c int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) · line 646
Quoted 10 lines
net/ipv4/tcp.c +4
-0@@ -636,6 +637,10 @@ int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) answ =READ_ONCE (tp->write_seq) -READ_ONCE (tp->snd_nxt);break ; +#if IS_ENABLED (CONFIG_NETGPU ) +case NETGPU_SOCK_IOCTL_ATTACH_QUEUES :/* SIOCPROTOPRIVATE */ +return netgpu_attach_socket (sk, (void __user *)arg); +#endif default :return -ENOIOCTLCMD ; }
Actually, this is just ugly, so I'm going to rip it out and have it done the other way around: (ctx -> sk) instead of (sk -> ctx), so ignore this.
Signature · 1 line
Jonathan
Greg KH
maintainerRe: [RFC PATCH v2 10/21] netgpu: add network/gpu/host dma module
In reply to Jonathan LemonQuoted 11 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
Netgpu provides a data path for zero-copy sends and receives without having the host CPU touch the data. Protocol processing is done on the host CPU, while data is DMA'd to and from DMA mapped memory areas. The initial code provides transfers between (mlx5 / host memory) and (mlx5 / nvidia GPU memory).
The use case for this module are GPUs used for machine learning, which are located near the NICs, and have a high bandwidth PCI connection between the GPU/NIC.
Do we have such a GPU driver in the kernel today? We can't add new apis/interfaces for no in-kernel users, as you well know.
There's lots of crazyness in this patch, but this is just really odd:
drivers/misc/netgpu/netgpu_main.c line 37
Quoted 5 lines
drivers/misc/netgpu/netgpu_main.c new file +5
-0+#if IS_MODULE (CONFIG_NETGPU ) +#define MAYBE_EXPORT_SYMBOL (s) +#else +#define MAYBE_EXPORT_SYMBOL (s) EXPORT_SYMBOL(s) +#endif
Why is that needed at all? Why does no one else in the kernel need such a thing?
And why EXPORT_SYMBOL() and not EXPORT_SYMBOL_GPL() (I have to ask).
thanks,
greg k-h
Greg KH
maintainerRe: [RFC PATCH v2 19/21] mlx5e: add the netgpu driver functions
In reply to Jonathan Lemondrivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c line 4
Quoted 7 lines · Jonathan Lemon
drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c new file +4
-0@@ -0,0 +1,340 @@ +#include "en.h" +#include "en/xdp.h" +#include "en/params.h" +#include "en/netgpu/setup.h"
<snip>
Always run scripts/checkpatch.pl on your patches do you do not get grumpy driver maintainers telling you to run scripts/checkpatch.pl on your patches.
greg k-h
Greg KH
maintainerRe: [RFC PATCH v2 11/21] core/skbuff: add page recycling logic for netgpu pages
In reply to Jonathan LemonQuoted 5 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
netgpu pages will always have a refcount of at least one (held by the netgpu module). If the skb is marked as containing netgpu ZC pages, recycle them back to netgpu.
What???
Quoted 24 lines
Signed-off-by Jonathan Lemon <[email protected]>--- net/core/skbuff.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)net/core/skbuff.c +10
-0@@ -591,6 +591,27 @@ static void skb_free_head(struct sk_buff *skb)kfree (head); } +#if IS_ENABLED (CONFIG_NETGPU ) +static void skb_netgpu_unref (struct skb_shared_info *shinfo) +{ +struct netgpu_ifq *ifq = shinfo->destructor_arg; +struct page *page; +int i; + +/* pages attached for skbs for TX shouldn't come here, since +* the skb is not marked as "zc_netgpu". (only RX skbs have this). +* dummy page does come here, but always has elevated refc.
net/core/skbuff.c static void skb_free_head(struct sk_buff *skb) · line 613
Quoted 10 lines
net/core/skbuff.c +10
-0+* +* Undelivered zc skb's will arrive at this point. +*/ +for (i =0 ; i < shinfo->nr_frags; i++) { + page =skb_frag_page (&shinfo->frags[i]); +if (page &&page_ref_dec_return (page) <=2 ) +netgpu_put_page (ifq, page, false); + } +} +#endif
Becides the basic "no #if in C files" issue here, why is this correct?
Quoted 9 lines
net/core/skbuff.c +1
-1+static void skb_release_data (struct sk_buff *skb) {struct skb_shared_info *shinfo =skb_shinfo (skb);@@ -601,8 +622,15 @@ static void skb_release_data(struct sk_buff *skb) &shinfo->dataref))return ; -for (i =0 ; i < shinfo->nr_frags; i++)
net/core/skbuff.c static void skb_release_data(struct sk_buff *skb) · line 633
Quoted 10 lines
net/core/skbuff.c +9
-1-__skb_frag_unref (&shinfo->frags[i]); +#if IS_ENABLED (CONFIG_NETGPU ) +if (skb->zc_netgpu && shinfo->nr_frags) { +skb_netgpu_unref (shinfo); + } else +#endif + { +for (i =0 ; i < shinfo->nr_frags; i++) +__skb_frag_unref (&shinfo->frags[i]); + }
Again, no #if in C code. But even then, this feels really really wrong.
greg k-h
Greg KH
maintainerRe: [RFC PATCH v2 21/21] netgpu/nvidia: add Nvidia plugin for netgpu
In reply to Jonathan LemonQuoted 8 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
This provides the interface between the netgpu core module and the nvidia kernel driver. This should be built as an external module, pointing to the nvidia build. For example:
export NV_PACKAGE_DIR=/w/nvidia/NVIDIA-Linux-x86_64-440.64 make -C ${kdir} M=
pwdO=obj $*
Ok, now you are just trolling us.
Nice job, I shouldn't have read the previous patches.
Please, go get a lawyer to sign-off on this patch, with their corporate email address on it. That's the only way we could possibly consider something like this.
Oh, and we need you to use your corporate email address too, as you are not putting copyright notices on this code, we will need to know who to come after in the future.
greg k-h
Re: [RFC PATCH v2 21/21] netgpu/nvidia: add Nvidia plugin for netgpu
Quoted 14 lines · Greg KH
Quoted 8 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
This provides the interface between the netgpu core module and the nvidia kernel driver. This should be built as an external module, pointing to the nvidia build. For example:
export NV_PACKAGE_DIR=/w/nvidia/NVIDIA-Linux-x86_64-440.64 make -C ${kdir} M=
pwdO=obj $*Ok, now you are just trolling us.
Nice job, I shouldn't have read the previous patches.
Quoted 7 lines
Please, go get a lawyer to sign-off on this patch, with their corporate email address on it. That's the only way we could possibly consider something like this.
Oh, and we need you to use your corporate email address too, as you are not putting copyright notices on this code, we will need to know who to come after in the future.
Jonathan, I think we need to do a better job talking about patches that are just meant to enable possible users vs patches that we actually hope the upstream kernel to take. Obviously code that only supports out of tree drivers isn’t a good fit for the upstream kernel. From the point of view of experimenting with these patches, GPUs benefit a lot from this functionality so I think it does make sense to have the enabling patches somewhere, just not in this series.
We’re finding it more common to have pcie switch hops between a [ GPU, NIC ] pair and the CPU, which gives a huge advantage to out of tree drivers or extensions that can DMA directly between the GPU/NIC without having to copy through the CPU. I’d love to have an alternative built on TCP because that’s where we invest the vast majority of our tuning, security and interoperability testing. It’s just more predictable overall.
This isn’t a new story, but if we can layer on APIs that enable this cleanly for in-tree drivers, we can work with the vendors to use better supported APIs and have a more stable kernel. Obviously this is an RFC and there’s a long road ahead, but as long as the upstream kernel doesn’t provide an answer, out of tree drivers are going to fill in the weak spots.
Other possible use cases would include also include other GPUs or my favorite:
NVME <-> filesystem <-> NIC with io_uring driving the IO and without copies.
-chris
Re: [RFC PATCH v2 21/21] netgpu/nvidia: add Nvidia plugin for netgpu
Quoted 9 lines · Chris Mason
Quoted 1 line
come after in the future.
Jonathan, I think we need to do a better job talking about patches that are just meant to enable possible users vs patches that we actually hope the upstream kernel to take. Obviously code that only supports out of tree drivers isn???t a good fit for the upstream kernel. From the point of view of experimenting with these patches, GPUs benefit a lot from this functionality so I think it does make sense to have the enabling patches somewhere, just not in this series.
Sorry, but his crap is built only for this use case, and that is what really pissed people off as it very much looks intentional.
Quoted 13 lines
We???re finding it more common to have pcie switch hops between a [ GPU, NIC ] pair and the CPU, which gives a huge advantage to out of tree drivers or extensions that can DMA directly between the GPU/NIC without having to copy through the CPU. I???d love to have an alternative built on TCP because that???s where we invest the vast majority of our tuning, security and interoperability testing. It???s just more predictable overall.
This isn???t a new story, but if we can layer on APIs that enable this cleanly for in-tree drivers, we can work with the vendors to use better supported APIs and have a more stable kernel. Obviously this is an RFC and there???s a long road ahead, but as long as the upstream kernel doesn???t provide an answer, out of tree drivers are going to fill in the weak spots.
Quoted 4 lines
Other possible use cases would include also include other GPUs or my favorite:
NVME <-> filesystem <-> NIC with
io_uringdriving the IO and without copies.
And we have all that working with the existing p2pdma infrastructure (at least if you're using RDMA insted of badly reinventing it, but it could be added to other users easily).
That infrastructure is EXPORT_SYMBOL_GPL as it should be for infrastructure like that, and a lot of his crap just seems to be because he's working around that.
So I really agree with Gred, this very much looks like a deliberate trolling attempt.
Re: [RFC PATCH v2 10/21] netgpu: add network/gpu/host dma module
In reply to Greg KHQuoted 9 lines · Greg KH
Quoted 8 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
Netgpu provides a data path for zero-copy sends and receives without having the host CPU touch the data. Protocol processing is done on the host CPU, while data is DMA'd to and from DMA mapped memory areas. The initial code provides transfers between (mlx5 / host memory) and (mlx5 / nvidia GPU memory).
Quoted 6 lines
Quoted 3 lines
The use case for this module are GPUs used for machine learning, which are located near the NICs, and have a high bandwidth PCI connection between the GPU/NIC.
Do we have such a GPU driver in the kernel today? We can't add new apis/interfaces for no in-kernel users, as you well know.
No, that's what I'm trying to create. But Jens pointed out that the main sticking point here seems to be Nvidia, so I'll look into seeing whether there are some AMD or Intel GPUS I can use.
Quoted 10 lines
There's lots of crazyness in this patch, but this is just really odd:
drivers/misc/netgpu/netgpu_main.c line 37
Quoted 5 lines
drivers/misc/netgpu/netgpu_main.c new file +5
-0+#if IS_MODULE (CONFIG_NETGPU ) +#define MAYBE_EXPORT_SYMBOL (s) +#else +#define MAYBE_EXPORT_SYMBOL (s) EXPORT_SYMBOL(s) +#endif Why is that needed at all? Why does no one else in the kernel need such a thing?
Really, this is just development code, allowing the netgpu to be built as a loadable module. I'll rip it out.
Quoted 1 line
And why
EXPORT_SYMBOL()and notEXPORT_SYMBOL_GPL()(I have to ask).
Shorter typing, didn't think to add _GPL, I'll do that.
Signature · 1 line
Jonathan
Re: [RFC PATCH v2 11/21] core/skbuff: add page recycling logic for netgpu pages
In reply to Greg KHQuoted 8 lines · Greg KH
Quoted 5 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
netgpu pages will always have a refcount of at least one (held by the netgpu module). If the skb is marked as containing netgpu ZC pages, recycle them back to netgpu.
What???
Yes, this is page refcount elevation. ZONE_DEVICE pages do this also, which is hidden in put_devmap_managed_page(). I would really like to find a generic solution for this, as it has come up in other cases as well (page recycling for page_pool, for example). Some way to say "this page is different", and a separate routine to release refcounts.
Can we have a discussion on this possibility?
Signature · 1 line
Jonathan
Greg KH
maintainerRe: [RFC PATCH v2 11/21] core/skbuff: add page recycling logic for netgpu pages
Quoted 10 lines · Jonathan Lemon
Quoted 8 lines · Greg KH
Quoted 5 lines · Jonathan Lemon
From: Jonathan Lemon <[email protected]>
netgpu pages will always have a refcount of at least one (held by the netgpu module). If the skb is marked as containing netgpu ZC pages, recycle them back to netgpu.
What???
Quoted 7 lines
Yes, this is page refcount elevation.
ZONE_DEVICEpages do this also, which is hidden input_devmap_managed_page(). I would really like to find a generic solution for this, as it has come up in other cases as well (page recycling forpage_pool, for example). Some way to say "this page is different", and a separate routine to release refcounts.Can we have a discussion on this possibility?
Then propose a generic solution, not a "solution" like this.
greg k-h
Re: [RFC PATCH v2 21/21] netgpu/nvidia: add Nvidia plugin for netgpu
In reply to Christoph HellwigQuoted 11 lines · Christoph Hellwig
Quoted 9 lines · Chris Mason
Quoted 1 line
come after in the future.
Jonathan, I think we need to do a better job talking about patches that are just meant to enable possible users vs patches that we actually hope the upstream kernel to take. Obviously code that only supports out of tree drivers isn???t a good fit for the upstream kernel. From the point of view of experimenting with these patches, GPUs benefit a lot from this functionality so I think it does make sense to have the enabling patches somewhere, just not in this series.
Quoted 2 lines
Sorry, but his crap is built only for this use case, and that is what really pissed people off as it very much looks intentional.
No, we’ve had workloads asking for better zero copy solutions for ages. The goal is to address both this specialized workload and the general case zero copy tx/rx.
-chris
kernel test robot
build botRe: [RFC PATCH v2 05/21] uapi/misc: add shqueue.h for shared queues
In reply to Jonathan Lemon[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net-next/master]
[also build test ERROR on linus/master v5.8-rc7]
[cannot apply to char-misc/char-misc-testing hnaz-linux-mm/master net/master ipvs/master next-20200728] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200728-064743 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 350e7ab92da8420fe97e1edf040db8c9bab750f8 config: x86_64-randconfig-a006-20200728 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project e57464151d4c4912a7ec4d6fd0920056b2f75c7c)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from <built-in>:1:
>> ./usr/include/misc/shqueue.h:31:19: error: implicit declaration of function 'READ_ONCE' [-Werror,-Wimplicit-function-declaration]
q->cached_cons = READ_ONCE(*q->cons);
^
>> ./usr/include/misc/shqueue.h:37:2: error: implicit declaration of function 'smp_mb' [-Werror,-Wimplicit-function-declaration]
smp_mb(); /* D, matches A */
^
>> ./usr/include/misc/shqueue.h:38:2: error: implicit declaration of function 'WRITE_ONCE' [-Werror,-Wimplicit-function-declaration]
WRITE_ONCE(*q->cons, q->cached_cons);
^
./usr/include/misc/shqueue.h:44:19: error: implicit declaration of function 'READ_ONCE' [-Werror,-Wimplicit-function-declaration]
q->cached_prod = READ_ONCE(*q->prod);
^
>> ./usr/include/misc/shqueue.h:45:2: error: implicit declaration of function 'smp_rmb' [-Werror,-Wimplicit-function-declaration]
smp_rmb(); /* C, matches B */
^
>> ./usr/include/misc/shqueue.h:50:2: error: implicit declaration of function 'smp_wmb' [-Werror,-Wimplicit-function-declaration]
smp_wmb(); /* B, matches C */
^
./usr/include/misc/shqueue.h:51:2: error: implicit declaration of function 'WRITE_ONCE' [-Werror,-Wimplicit-function-declaration]
WRITE_ONCE(*q->prod, q->cached_prod);
^
>> ./usr/include/misc/shqueue.h:60:19: error: unknown type name 'bool'
static __inline__ bool sq_is_empty(struct shared_queue *q)
^
./usr/include/misc/shqueue.h:62:9: error: implicit declaration of function 'READ_ONCE' [-Werror,-Wimplicit-function-declaration]
return READ_ONCE(*q->prod) == READ_ONCE(*q->cons);
^
./usr/include/misc/shqueue.h:65:19: error: unknown type name 'bool'
static __inline__ bool sq_cons_empty(struct shared_queue *q)
^
./usr/include/misc/shqueue.h:83:19: error: unknown type name 'bool'
static __inline__ bool sq_cons_avail(struct shared_queue *q, unsigned count)
^
>> ./usr/include/misc/shqueue.h:86:10: error: use of undeclared identifier 'true'
return true;
^
>> ./usr/include/misc/shqueue.h:106:11: error: use of undeclared identifier 'NULL'
return NULL;
^
./usr/include/misc/shqueue.h:174:19: error: unknown type name 'bool'
static __inline__ bool sq_prod_avail(struct shared_queue *q, unsigned count)
^
./usr/include/misc/shqueue.h:177:10: error: use of undeclared identifier 'true'
return true;
^
./usr/include/misc/shqueue.h:190:10: error: use of undeclared identifier 'NULL'
return NULL;
^16 errors generated.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Quoted 6 lines · On Mon, 27 Jul 2020 15:44:23 -0700 Jonathan Lemon <[email protected]>…
Current limitations:
- mlx5 only, header splitting is at a fixed offset.
- currently only TCP protocol delivery is performed.
- TX completion notification is planned, but not in this patchset.
- not compatible with xsk (re-uses same datastructures)
- not compatible with bpf payload inspection
This a good summary of why TCP Offload is not a mainstream solution. Look back in archives and you will find lots of presentations about why TOE sucks.
You also forgot no VRF, no namespaes, no firewall, no containers, no encapsulation. It acts as proof that if you cut out everything you can build something faster. But not suitable for upstream or production.
Quoted 10 lines · Stephen Hemminger
Quoted 6 lines · On Mon, 27 Jul 2020 15:44:23 -0700 Jonathan Lemon <[email protected]>…
Current limitations:
- mlx5 only, header splitting is at a fixed offset.
- currently only TCP protocol delivery is performed.
- TX completion notification is planned, but not in this patchset.
- not compatible with xsk (re-uses same datastructures)
- not compatible with bpf payload inspection
Quoted 3 lines
This a good summary of why TCP Offload is not a mainstream solution. Look back in archives and you will find lots of presentations about why TOE sucks.
I.. agree with this? But I'm failing to see what TCP offload (or any HW offload) has to do with the change. I'm trying to do the opposite of HW offload here - keeping the protocol in the kernel. Although obviously what I'm doing is not suitable for all use cases.
Quoted 4 lines
You also forgot no VRF, no namespaes, no firewall, no containers, no encapsulation. It acts as proof that if you cut out everything you can build something faster. But not suitable for upstream or production.
Signature · 1 line
Jonathan
kernel test robot
build botRe: [RFC PATCH v2 06/21] include: add netgpu UAPI and kernel definitions
In reply to Jonathan Lemon[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net-next/master]
[also build test ERROR on linus/master v5.8-rc7]
[cannot apply to char-misc/char-misc-testing hnaz-linux-mm/master net/master ipvs/master next-20200728] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200728-064743 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 350e7ab92da8420fe97e1edf040db8c9bab750f8 config: x86_64-randconfig-a006-20200728 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project e57464151d4c4912a7ec4d6fd0920056b2f75c7c)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from <built-in>:1:
>> ./usr/include/misc/netgpu.h:35:15: error: field has incomplete type 'struct iovec'
struct iovec iov;
^
./usr/include/misc/netgpu.h:35:9: note: forward declaration of 'struct iovec'
struct iovec iov;
^1 error generated.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org