netgpu: networking between NIC and GPU/CPU.
Subsystems (15)
cover netgpu: networking between NIC and GPU/CPU.
This series is a working RFC proof-of-concept that implements DMA zero-copy between the NIC and a GPU device for the data path, while keeping the protocol processing on the host CPU.
This also works for zero-copy send/recv to host (CPU) memory.
Current limitations:
- mlx5 only, header splitting is at a fixed offset.
- currently only TCP protocol delivery is performed.
- not optimized (hey, it works!)
- TX completion notification is planned, but not in this patchset.
- one socket per device
- not compatible with xsk (re-uses same datastructures)
- not compatible with bpf payload inspection
- x86 !iommu only; liberties are taken with PA addresses.
The next section provides a brief overview of how things work, for this phase 0 proof of concept.
A transport context is created on a device, which sets up the datapath, and the device queues. Only specialized RX queues are needed, the standard TX queues are used for packet transmission.
Memory areas which participate in zero-copy transmission are registered with the context. These areas can be used as either RX packet buffers or TX data areas (or both). The memory can come from either malloc/mmap or cudaMalloc(). The latter call provides a handle to the userspace application, but the memory region is only accessible to the GPU.
A socket is created and registered with the context, which sets SOCK_ZEROCOPY, and is bound to the device with SO_BINDTODEVICE.
Asymmetrical data paths are possible (zc TX, normal RX), and vice versa, but the curreent PoC sets things up for symmetrical transport. The application needs to provide the RX buffers to the receive queue, similar to AF_XDP.
Once things are set up, data is sent to the network with sendmsg(). The iovecs provided contain an address in the region previously registered. The normal protocol stack processing constructs the packet, but the data is not touched by the stack. In this phase, the application is not notified when the protocol processing is complete and the data area is safe to modify again.
For RX, packets undergo the usual protocol processing and are delivered up to the socket receive queue. At this point, the skb data fragments are delivered to the application as iovecs through an AF_XDP style queue. The application can poll for readability, but does not use read() to receive the data.
The initial application used is iperf3, a modified version with the userspace library is available at:
https://github.com/jlemon/iperf
https://github.com/jlemon/netgpu
Running "iperf3 -s -z --dport 8888" (host memory) on a 12Gbps link:
11.3 Gbit/sec receive
10.8 Gbit/sec tramsmit
Running "iperf3 -s -z --dport 8888 --gpu" on a 25Gbps link:
22.5 Gbit/sec receive
12.6 Gbit/sec transmit (!!!)
For the GPU runs, the Intel PCI monitoring tools were used to confirm that the host PCI bus was mostly idle. The TX performance needs further investigation.
Comments welcome. The next phase of the work will clean up the interface, adding completion notifications, and a flexible queue creation mechanism.
--
Jonathan
Jonathan Lemon (21):
mm: add {add|release}_memory_pages
mm: Allow DMA mapping of pages which are not online
tcp: Pad TCP options out to a fixed size
mlx5: add definitions for header split and netgpu mlx5/xsk: check that xsk does not conflict with netgpu
mlx5: add header_split flag
mlx5: remove the umem parameter from mlx5e_open_channel
misc: add shqueue.h for prototyping
include: add definitions for netgpu
mlx5: add netgpu queue functions
skbuff: add a zc_netgpu bitflag
mlx5: hook up the netgpu channel functions
netdevice: add SETUP_NETGPU to the netdev_bpf structure
kernel: export free_uid
netgpu: add network/gpu dma module
lib: have __zerocopy_sg_from_iter get netgpu pages for a sk net/core: add the SO_REGISTER_DMA socket option
tcp: add MSG_NETDMA flag for sendmsg()
core: add page recycling logic for netgpu pages core/skbuff: use skb_zdata for testing whether skb is zerocopy
mlx5: add XDP_SETUP_NETGPU hook
drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/netgpu/Kconfig | 10 + drivers/misc/netgpu/Makefile | 11 + drivers/misc/netgpu/nvidia.c | 1516 +++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/Makefile | 3 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 22 +- .../mellanox/mlx5/core/en/netgpu/setup.c | 475 ++++++ .../mellanox/mlx5/core/en/netgpu/setup.h | 42 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 + .../ethernet/mellanox/mlx5/core/en/xsk/umem.c | 3 + .../ethernet/mellanox/mlx5/core/en/xsk/umem.h | 3 + .../ethernet/mellanox/mlx5/core/en_ethtool.c | 15 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 118 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 52 +- .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 15 +- include/linux/dma-mapping.h | 4 +- include/linux/memory_hotplug.h | 4 + include/linux/mmzone.h | 7 + include/linux/netdevice.h | 6 + include/linux/skbuff.h | 27 +- include/linux/socket.h | 1 + include/linux/uio.h | 4 + include/net/netgpu.h | 65 + include/uapi/asm-generic/socket.h | 2 + include/uapi/misc/netgpu.h | 43 + include/uapi/misc/shqueue.h | 205 +++ kernel/user.c | 1 + lib/iov_iter.c | 45 + mm/memory_hotplug.c | 65 +- net/core/datagram.c | 6 +- net/core/skbuff.c | 44 +- net/core/sock.c | 26 + net/ipv4/tcp.c | 8 + net/ipv4/tcp_output.c | 16 + 35 files changed, 2828 insertions(+), 41 deletions(-) create mode 100644 drivers/misc/netgpu/Kconfig create mode 100644 drivers/misc/netgpu/Makefile create mode 100644 drivers/misc/netgpu/nvidia.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.h create mode 100644 include/net/netgpu.h create mode 100644 include/uapi/misc/netgpu.h create mode 100644 include/uapi/misc/shqueue.h
Signature · 1 line
2.24.1
RFC PATCH 1/21 mm: add {add|release}_memory_pages
This allows creation of system pages at a specific physical address, which is useful for creating dummy backing pages which correspond to unaddressable external memory at specific locations.
--- include/linux/memory_hotplug.h | 4 +++ mm/memory_hotplug.c | 65 ++++++++++++++++++++++++++++++++-- 2 files changed, 67 insertions(+), 2 deletions(-)
include/linux/memory_hotplug.h +4 -0
@@ -138,6 +138,10 @@ extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,extern int __add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages,struct mhp_params *params); +struct resource *add_memory_pages (int nid,u64 start,u64 size, +struct mhp_params *params); +void release_memory_pages (struct resource *res); +#ifndef CONFIG_ARCH_HAS_ADD_PAGES static inline int add_pages (int nid,unsigned long start_pfn,unsigned long nr_pages,struct mhp_params *params)
mm/memory_hotplug.c +63 -2
77 lines not on this page · raw message
Signature · 1 line
2.24.1
RFC PATCH 2/21 mm: Allow DMA mapping of pages which are not online
In reply to Jonathan LemonChange the system RAM check from 'valid' to 'online', so dummy pages which refer to external DMA resources can be mapped.
--- include/linux/dma-mapping.h | 4 ++-- include/linux/mmzone.h | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
include/linux/dma-mapping.h +2 -2
@@ -348,8 +348,8 @@ static inline dma_addr_t dma_map_resource(struct device *dev,BUG_ON (!valid_dma_direction (dir)); -/* Don't allow RAM to be mapped */ -if (WARN_ON_ONCE (pfn_valid (PHYS_PFN (phys_addr)))) +/* Don't allow online RAM to be mapped */ +if (WARN_ON_ONCE (pfn_online (PHYS_PFN (phys_addr))))return DMA_MAPPING_ERROR ;if (dma_is_direct (ops))
include/linux/mmzone.h +7 -0
@@ -1348,6 +1348,13 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr)return -1 ; } +static inline int pfn_online (unsigned long pfn) +{ +if (pfn_to_section_nr (pfn) >=NR_MEM_SECTIONS ) +return 0 ; +return online_section (__nr_to_section (pfn_to_section_nr (pfn))); +} +/* * These are _only_ used during initialisation, therefore they * can use __initdata ... They could have names to indicate
Signature · 1 line
2.24.1
kernel test robot
build botRe: [RFC PATCH 02/21] mm: Allow DMA mapping of pages which are not online
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net/master]
[also build test ERROR on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: i386-tinyconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/thread_info.h:12,
from arch/x86/include/asm/preempt.h:7,
from include/linux/preempt.h:78,
from include/linux/spinlock.h:51,
from include/linux/seqlock.h:36,
from include/linux/time.h:6,
from include/linux/stat.h:19,
from include/linux/module.h:13,
from init/do_mounts.c:2:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~ cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5, from include/linux/mm.h:9,
from kernel/exit.c:8:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
kernel/exit.c: At top level:
kernel/exit.c:1714:13: warning: no previous prototype for 'abort' [-Wmissing-prototypes]
1714 | __weak void abort(void)
| ^~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/jump_label.h:250,
from include/linux/static_key.h:1,
from arch/x86/include/asm/nospec-branch.h:6,
from arch/x86/include/asm/irqflags.h:9,
from include/linux/irqflags.h:16,
from include/linux/rcupdate.h:26,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from kernel/sched/sched.h:5,
from kernel/sched/core.c:9:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
kernel/sched/core.c: In function 'ttwu_stat':
kernel/sched/core.c:2160:13: warning: variable 'rq' set but not used [-Wunused-but-set-variable]
2160 | struct rq *rq;
| ^~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/jump_label.h:250,
from include/linux/static_key.h:1,
from arch/x86/include/asm/nospec-branch.h:6,
from arch/x86/include/asm/irqflags.h:9,
from include/linux/irqflags.h:16,
from include/linux/rcupdate.h:26,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from kernel/sched/sched.h:5,
from kernel/sched/fair.c:23:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
kernel/sched/fair.c: At top level:
kernel/sched/fair.c:5352:6: warning: no previous prototype for 'init_cfs_bandwidth' [-Wmissing-prototypes]
5352 | void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b) {}
| ^~~~~~~~~~~~~~~~~~
kernel/sched/fair.c:11078:6: warning: no previous prototype for 'free_fair_sched_group' [-Wmissing-prototypes]
11078 | void free_fair_sched_group(struct task_group *tg) { }
| ^~~~~~~~~~~~~~~~~~~~~
kernel/sched/fair.c:11080:5: warning: no previous prototype for 'alloc_fair_sched_group' [-Wmissing-prototypes]
11080 | int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
| ^~~~~~~~~~~~~~~~~~~~~~
kernel/sched/fair.c:11085:6: warning: no previous prototype for 'online_fair_sched_group' [-Wmissing-prototypes]
11085 | void online_fair_sched_group(struct task_group *tg) { }
| ^~~~~~~~~~~~~~~~~~~~~~~
kernel/sched/fair.c:11087:6: warning: no previous prototype for 'unregister_fair_sched_group' [-Wmissing-prototypes]
11087 | void unregister_fair_sched_group(struct task_group *tg) { }
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/jump_label.h:250,
from include/linux/static_key.h:1,
from arch/x86/include/asm/nospec-branch.h:6,
from arch/x86/include/asm/irqflags.h:9,
from include/linux/irqflags.h:16,
from include/linux/rcupdate.h:26,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from kernel/sched/sched.h:5,
from kernel/sched/rt.c:6:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
kernel/sched/rt.c: At top level:
kernel/sched/rt.c:253:6: warning: no previous prototype for 'free_rt_sched_group' [-Wmissing-prototypes]
253 | void free_rt_sched_group(struct task_group *tg) { }
| ^~~~~~~~~~~~~~~~~~~ kernel/sched/rt.c:255:5: warning: no previous prototype for 'alloc_rt_sched_group' [-Wmissing-prototypes]
255 | int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
| ^~~~~~~~~~~~~~~~~~~~
kernel/sched/rt.c:668:6: warning: no previous prototype for 'sched_rt_bandwidth_account' [-Wmissing-prototypes]
668 | bool sched_rt_bandwidth_account(struct rt_rq *rt_rq)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/thread_info.h:12,
from arch/x86/include/asm/preempt.h:7,
from include/linux/preempt.h:78,
from include/linux/spinlock.h:51,
from include/linux/wait.h:9,
from include/linux/wait_bit.h:8,
from include/linux/fs.h:6,
from kernel/events/core.c:11:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~ kernel/events/core.c: At top level:
kernel/events/core.c:6490:6: warning: no previous prototype for 'perf_pmu_snapshot_aux' [-Wmissing-prototypes]
6490 | long perf_pmu_snapshot_aux(struct perf_buffer *rb,
| ^~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/mm.h:9,
from arch/x86/kernel/doublefault_32.c:2:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~ arch/x86/kernel/doublefault_32.c: At top level:
arch/x86/kernel/doublefault_32.c:23:36: warning: no previous prototype for 'doublefault_shim' [-Wmissing-prototypes]
23 | asmlinkage noinstr void __noreturn doublefault_shim(void)
| ^~~~~~~~~~~~~~~~
arch/x86/kernel/doublefault_32.c:119:6: warning: no previous prototype for 'doublefault_init_cpu_tss' [-Wmissing-prototypes]
119 | void doublefault_init_cpu_tss(void)
| ^~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/gfp.h:5,
from include/linux/slab.h:15,
from arch/x86/kernel/cpu/cacheinfo.c:11:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
arch/x86/kernel/cpu/cacheinfo.c: In function 'init_intel_cacheinfo':
arch/x86/kernel/cpu/cacheinfo.c:727:26: warning: variable 'l3_id' set but not used [-Wunused-but-set-variable]
727 | unsigned int l2_id = 0, l3_id = 0, num_threads_sharing, index_msb;
| ^~~~~
arch/x86/kernel/cpu/cacheinfo.c:727:15: warning: variable 'l2_id' set but not used [-Wunused-but-set-variable]
727 | unsigned int l2_id = 0, l3_id = 0, num_threads_sharing, index_msb;
| ^~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/jump_label.h:250,
from include/linux/static_key.h:1,
from arch/x86/include/asm/nospec-branch.h:6,
from arch/x86/include/asm/irqflags.h:9,
from include/linux/irqflags.h:16,
from include/linux/rcupdate.h:26,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from include/linux/audit.h:12,
from security/commoncap.c:6:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
In file included from include/linux/capability.h:16,
from security/commoncap.c:5:
security/commoncap.c: In function 'cap_prctl_drop':
include/uapi/linux/capability.h:413:27: warning: comparison of unsigned expression >= 0 is always true [-Wtype-limits]
413 | #define cap_valid(x) ((x) >= 0 && (x) <= CAP_LAST_CAP)
| ^~
security/commoncap.c:1147:7: note: in expansion of macro 'cap_valid'
1147 | if (!cap_valid(cap))
| ^~~~~~~~~
security/commoncap.c: In function 'cap_task_prctl':
include/uapi/linux/capability.h:413:27: warning: comparison of unsigned expression >= 0 is always true [-Wtype-limits]
413 | #define cap_valid(x) ((x) >= 0 && (x) <= CAP_LAST_CAP)
| ^~
security/commoncap.c:1177:8: note: in expansion of macro 'cap_valid'
1177 | if (!cap_valid(arg2))
| ^~~~~~~~~
include/uapi/linux/capability.h:413:27: warning: comparison of unsigned expression >= 0 is always true [-Wtype-limits]
413 | #define cap_valid(x) ((x) >= 0 && (x) <= CAP_LAST_CAP)
| ^~
security/commoncap.c:1262:10: note: in expansion of macro 'cap_valid'
1262 | if (((!cap_valid(arg3)) | arg4 | arg5))
| ^~~~~~~~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/cpumask.h:14,
from arch/x86/include/asm/cpumask.h:5,
from arch/x86/include/asm/msr.h:11,
from arch/x86/include/asm/processor.h:22,
from include/linux/mutex.h:19,
from include/linux/notifier.h:14,
from include/linux/clk.h:14,
from lib/vsprintf.c:22:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
lib/vsprintf.c: In function 'va_format':
lib/vsprintf.c:1660:2: warning: function 'va_format' might be a candidate for 'gnu_printf' format attribute [-Wsuggest-attribute=format]
1660 | buf += vsnprintf(buf, end > buf ? end - buf : 0, va_fmt->fmt, va);
| ^~~
cc1: some warnings being treated as errors
--
In file included from arch/x86/include/asm/bug.h:86,
from include/linux/bug.h:5,
from include/linux/jump_label.h:250,
from include/linux/static_key.h:1,
from arch/x86/include/asm/nospec-branch.h:6,
from arch/x86/include/asm/irqflags.h:9,
from include/linux/irqflags.h:16,
from include/linux/rcupdate.h:26,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from include/linux/ratelimit.h:6,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/linux/platform_device.h:13,
from drivers/base/platform.c:13:
include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: error: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Werror=implicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~
include/asm-generic/bug.h:187:25: note: in definition of macro 'WARN_ON'
187 | int __ret_warn_on = !!(condition); \
| ^~~~~~~~~
include/linux/dma-mapping.h:352:6: note: in expansion of macro 'WARN_ON_ONCE'
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~~~
drivers/base/platform.c: At top level:
drivers/base/platform.c:1352:20: warning: no previous prototype for 'early_platform_cleanup' [-Wmissing-prototypes]
1352 | void __weak __init early_platform_cleanup(void) { }
| ^~~~~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +352 include/linux/dma-mapping.h
339
340 static inline dma_addr_t dma_map_resource(struct device *dev,
341 phys_addr_t phys_addr,
342 size_t size,
343 enum dma_data_direction dir,
344 unsigned long attrs)
345 {
346 const struct dma_map_ops *ops = get_dma_ops(dev);
347 dma_addr_t addr = DMA_MAPPING_ERROR;
348
349 BUG_ON(!valid_dma_direction(dir));
350
351 /* Don't allow online RAM to be mapped */
> 352 if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr)))) 353 return DMA_MAPPING_ERROR;
354
355 if (dma_is_direct(ops))
356 addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs);
357 else if (ops->map_resource)
358 addr = ops->map_resource(dev, phys_addr, size, dir, attrs);
359
360 debug_dma_map_resource(dev, phys_addr, size, dir, addr);
361 return addr;
362 }
363
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 02/21] mm: Allow DMA mapping of pages which are not online
In reply to Jonathan Lemon[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on net/master]
[also build test WARNING on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: i386-allyesconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>, old ones prefixed by <<):
In file included from arch/x86/include/asm/bug.h:86, from include/linux/bug.h:5,
from include/linux/thread_info.h:12,
from arch/x86/include/asm/preempt.h:7,
from include/linux/preempt.h:78,
from include/linux/spinlock.h:51,
from include/linux/seqlock.h:36,
from include/linux/time.h:6,
from include/linux/efi.h:17,
from drivers/firmware/efi/libstub/x86-stub.c:9:include/linux/dma-mapping.h: In function 'dma_map_resource':
>> include/linux/dma-mapping.h:352:19: warning: implicit declaration of function 'pfn_online'; did you mean 'cpu_online'? [-Wimplicit-function-declaration]
352 | if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr))))
| ^~~~~~~~~~include/asm-generic/bug.h:101:25: note: in definition of macro 'WARN_ON_ONCE'
101 | int __ret_warn_on = !!(condition); | ^~~~~~~~~
vim +352 include/linux/dma-mapping.h
339
340 static inline dma_addr_t dma_map_resource(struct device *dev,
341 phys_addr_t phys_addr,
342 size_t size,
343 enum dma_data_direction dir,
344 unsigned long attrs)
345 {
346 const struct dma_map_ops *ops = get_dma_ops(dev);
347 dma_addr_t addr = DMA_MAPPING_ERROR;
348
349 BUG_ON(!valid_dma_direction(dir));
350
351 /* Don't allow online RAM to be mapped */
> 352 if (WARN_ON_ONCE(pfn_online(PHYS_PFN(phys_addr)))) 353 return DMA_MAPPING_ERROR;
354
355 if (dma_is_direct(ops))
356 addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs);
357 else if (ops->map_resource)
358 addr = ops->map_resource(dev, phys_addr, size, dir, attrs);
359
360 debug_dma_map_resource(dev, phys_addr, size, dir, addr);
361 return addr;
362 }
363
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 3/21 tcp: Pad TCP options out to a fixed size
In reply to Jonathan LemonThe "header splitting" feature used by netgpu doesn't actually parse the incoming packet header. Instead, it splits the packet at a fixed offset. In order for this to work, the sender needs to send packets with a fixed header size.
(Obviously not for upstream committing, just for prototyping)
--- net/ipv4/tcp_output.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
net/ipv4/tcp_output.c +16 -0
@@ -438,6 +438,7 @@ struct tcp_out_options {u8 ws;/* window scale, 0 to disable */ u8 num_sack_blocks;/* number of SACK blocks to include */ u8 hash_size;/* bytes in hash_location */ +u8 pad_size;/* additional nops for padding */ __u8 *hash_location;/* temporary pointer, overloaded */ __u32 tsval, tsecr;/* need to include OPTION_TS */ struct tcp_fastopen_cookie *fastopen_cookie;/* Fast open cookie */
@@ -562,6 +563,15 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,smc_options_write (ptr, &options);mptcp_options_write (ptr, opts); + +/* pad out options for netgpu */ +if (opts->pad_size) { +int len = opts->pad_size; +u8 *p = (u8 *)ptr; + +while (len--) + *p++ =TCPOPT_NOP ; + } } static void smc_set_option(const struct tcp_sock *tp,
@@ -824,6 +834,12 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb opts->num_sack_blocks *TCPOLEN_SACK_PERBLOCK ; } +/* force padding for netgpu */ +if (size <20 ) { + opts->pad_size =20 - size; + size += opts->pad_size; + } +return size; }
Signature · 1 line
2.24.1
RFC PATCH 4/21 mlx5: add definitions for header split and netgpu
In reply to Jonathan LemonAdd definitions for fixed-length header splitting at TOTAL_HEADERS, and pointers for the upcoming netdma work. This reuses the same structures as xsk, so both cannot be run simultaneously.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 22 +++++++++++++++++-- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 +++ 2 files changed, 23 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +20 -2
@@ -58,6 +58,12 @@extern const struct net_device_ops mlx5e_netdev_ops;struct page_pool ; +#define TCP_HDRS_LEN (20 + 20)/* headers + options */ +#define IP6_HDRS_LEN (40) +#define MAC_HDR_LEN (14) +#define TOTAL_HEADERS (TCP_HDRS_LEN + IP6_HDRS_LEN + MAC_HDR_LEN) +#define HD_SPLIT_DEFAULT_FRAG_SIZE (4096) +#define MLX5E_HD_SPLIT (params) (params->hd_split)#define MLX5E_METADATA_ETHER_TYPE (0x8CE4)#define MLX5E_METADATA_ETHER_LEN 8
@@ -228,6 +234,7 @@ enum mlx5e_priv_flag {MLX5E_PFLAG_RX_STRIDING_RQ ,MLX5E_PFLAG_RX_NO_CSUM_COMPLETE ,MLX5E_PFLAG_XDP_TX_MPWQE , +MLX5E_PFLAG_RX_HD_SPLIT , MLX5E_NUM_PFLAGS, /* Keep last */ };
@@ -263,6 +270,7 @@ struct mlx5e_params {struct mlx5e_xsk *xsk;unsigned int sw_mtu;int hard_mtu; +bool hd_split; };enum {
@@ -299,7 +307,8 @@ struct mlx5e_cq_decomp {enum mlx5e_dma_map_type {MLX5E_DMA_MAP_SINGLE , -MLX5E_DMA_MAP_PAGE +MLX5E_DMA_MAP_PAGE , +MLX5E_DMA_MAP_FIXED };struct mlx5e_sq_dma {
@@ -367,6 +376,7 @@ struct mlx5e_dma_info {struct page *page;struct xdp_buff *xsk; }; +bool netgpu_source; };/* XDP packets can be transmitted in different ways. On completion, we need to
@@ -545,6 +555,7 @@ enum mlx5e_rq_flag {struct mlx5e_rq_frag_info {int frag_size;int frag_stride; +int frag_source; };struct mlx5e_rq_frags_info {
@@ -554,6 +565,7 @@ struct mlx5e_rq_frags_info {u8 wqe_bulk; }; +struct netgpu_ctx ; struct mlx5e_rq { /* data path */ union {
@@ -611,6 +623,7 @@ struct mlx5e_rq {/* AF_XDP zero-copy */ struct xdp_umem *umem; +struct netgpu_ctx *netgpu;struct work_struct recover_work;
@@ -628,6 +641,7 @@ struct mlx5e_rq {enum mlx5e_channel_state {MLX5E_CHANNEL_STATE_XSK , +MLX5E_CHANNEL_STATE_NETGPU ,MLX5E_CHANNEL_NUM_STATES };
@@ -736,9 +750,13 @@ struct mlx5e_xsk {* but it doesn't distinguish between zero-copy and non-zero-copy UMEMs, * so rely on our mechanism. */ -struct xdp_umem **umems; +union { +struct xdp_umem **umems; +struct netgpu_ctx **ctx_tbl; + };u16 refcnt;bool ever_used; +bool is_netgpu; };/* Temporary storage for variables that are allocated when struct mlx5e_priv is
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +3 -0
@@ -238,6 +238,9 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)case MLX5E_DMA_MAP_PAGE :dma_unmap_page (pdev, dma->addr, dma->size,DMA_TO_DEVICE );break ; +case MLX5E_DMA_MAP_FIXED : +/* DMA mappings are fixed, or managed elsewhere. */ +break ;default :WARN_ONCE (true,"mlx5e_tx_dma_unmap unknown DMA type!\n" ); }
Signature · 1 line
2.24.1
RFC PATCH 5/21 mlx5/xsk: check that xsk does not conflict with netgpu
In reply to Jonathan Lemonnetgpu will use the same data structures as xsk, so make sure that they are not conflicting.
--- drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c | 3 +++ drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h | 3 +++ 2 files changed, 6 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c +3 -0
@@ -27,7 +27,10 @@ static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk) sizeof(*xsk->umems), GFP_KERNEL);if (unlikely (!xsk->umems))return -ENOMEM ; + xsk->is_netgpu = false; } +if (xsk->is_netgpu) +return -EINVAL ; xsk->refcnt++; xsk->ever_used = true;
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h +3 -0
@@ -15,6 +15,9 @@ static inline struct xdp_umem *mlx5e_xsk_get_umem(struct mlx5e_params *params,if (unlikely (ix >= params->num_channels))return NULL ; +if (unlikely (xsk->is_netgpu)) +return NULL ; +return xsk->umems[ix]; }
Signature · 1 line
2.24.1
RFC PATCH 6/21 mlx5: add header_split flag
In reply to Jonathan LemonAdds a "rx_hd_split" private flag parameter to ethtool.
This enables header splitting, and sets up the fragment mappings. The feature is currently only enabled for netgpu channels.
--- .../ethernet/mellanox/mlx5/core/en_ethtool.c | 15 +++++++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 45 +++++++++++++++---- 2 files changed, 52 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +15 -0
@@ -1905,6 +1905,20 @@ static int set_pflag_xdp_tx_mpwqe(struct net_device *netdev, bool enable)return err; } +static int set_pflag_rx_hd_split (struct net_device *netdev,bool enable) +{ +struct mlx5e_priv *priv =netdev_priv (netdev); +int err; + + priv->channels.params.hd_split = enable; + err =mlx5e_safe_reopen_channels (priv); +if (err) +netdev_err (priv->netdev, +"%s failed to reopen channels, err(%d).\n" , + __func__, err); +return err; +} +static const struct pflag_desc mlx5e_priv_flags[MLX5E_NUM_PFLAGS ] = { {"rx_cqe_moder" , set_pflag_rx_cqe_based_moder }, {"tx_cqe_moder" , set_pflag_tx_cqe_based_moder },
@@ -1912,6 +1926,7 @@ static const struct pflag_desc mlx5e_priv_flags[MLX5E_NUM_PFLAGS] = { { "rx_striding_rq", set_pflag_rx_striding_rq }, { "rx_no_csum_complete", set_pflag_rx_no_csum_complete }, { "xdp_tx_mpwqe", set_pflag_xdp_tx_mpwqe }, + { "rx_hd_split", set_pflag_rx_hd_split }, }; static int mlx5e_handle_pflag(struct net_device *netdev,
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +37 -8
112 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [RFC PATCH 06/21] mlx5: add header_split flag
Quoted 4 lines · Jonathan Lemon
Adds a "
rx_hd_split" private flag parameter to ethtool.This enables header splitting, and sets up the fragment mappings. The feature is currently only enabled for netgpu channels.
We are using a similar idea (pseudo header split) to implement 4096+(headers) MTU at Google, to enable TCP RX zerocopy on x86.
Patch for mlx4 has not been sent upstream yet.
For mlx4, we are using a single buffer of 128*(number_of_slots_per_RX_RING), and 86 bytes for the first frag, so that the payload exactly fits a 4096 bytes page.
(In our case, most of our data TCP packets only have 12 bytes of TCP options)
I suggest that instead of a flag, you use a tunable, that can be set by ethtool, so that the exact number of bytes can be tuned, instead of hard coded in the driver.
(Patch for the counter part of [1] was resent 10 days ago on netdev@ by Govindarajulu Varadarajan) (Not sure if this has been merged yet)
[1]
commit f0db9b073415848709dd59a6394969882f517da9
Author: Govindarajulu Varadarajan <[email protected]> Date: Wed Sep 3 03:17:20 2014 +0530
ethtool: Add generic options for tunables
This patch adds new ethtool cmd, ETHTOOL_GTUNABLE & ETHTOOL_STUNABLE for getting tunable values from driver.
Add get_tunable and set_tunable to ethtool_ops. Driver implements these functions for getting/setting tunable value.
Quoted 166 lines
Signed-off-by Jonathan Lemon <[email protected]>--- .../ethernet/mellanox/mlx5/core/en_ethtool.c | 15 +++++++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 45 +++++++++++++++---- 2 files changed, 52 insertions(+), 8 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +15
-0@@ -1905,6 +1905,20 @@ static int set_pflag_xdp_tx_mpwqe(struct net_device *netdev, bool enable)return err; } +static int set_pflag_rx_hd_split (struct net_device *netdev,bool enable) +{ +struct mlx5e_priv *priv =netdev_priv (netdev); +int err; + + priv->channels.params.hd_split = enable; + err =mlx5e_safe_reopen_channels (priv); +if (err) +netdev_err (priv->netdev, +"%s failed to reopen channels, err(%d).\n" , + __func__, err); +return err; +} +static const struct pflag_desc mlx5e_priv_flags[MLX5E_NUM_PFLAGS ] = { {"rx_cqe_moder" , set_pflag_rx_cqe_based_moder }, {"tx_cqe_moder" , set_pflag_tx_cqe_based_moder },@@ -1912,6 +1926,7 @@ static const struct pflag_desc mlx5e_priv_flags[MLX5E_NUM_PFLAGS] = { { "rx_striding_rq", set_pflag_rx_striding_rq }, { "rx_no_csum_complete", set_pflag_rx_no_csum_complete }, { "xdp_tx_mpwqe", set_pflag_xdp_tx_mpwqe }, + { "rx_hd_split", set_pflag_rx_hd_split }, }; static int mlx5e_handle_pflag(struct net_device *netdev,drivers/net/ethernet/mellanox/mlx5/core/en_main.c +37
-8@@ -123,7 +123,8 @@ bool mlx5e_striding_rq_possible(struct mlx5_core_dev *mdev,void mlx5e_set_rq_type (struct mlx5_core_dev *mdev,struct mlx5e_params *params) { - params->rq_wq_type =mlx5e_striding_rq_possible (mdev, params) && + params->rq_wq_type =MLX5E_HD_SPLIT (params) ?MLX5_WQ_TYPE_CYCLIC : +mlx5e_striding_rq_possible (mdev, params) &&MLX5E_GET_PFLAG (params,MLX5E_PFLAG_RX_STRIDING_RQ ) ?MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ :MLX5_WQ_TYPE_CYCLIC ;@@ -323,6 +324,8 @@ static void mlx5e_init_frags_partition(struct mlx5e_rq *rq)if (prev) prev->last_in_page = true; } + next_frag.di->netgpu_source = + !!frag_info[f].frag_source; *frag = next_frag;/* prepare next */ @@ -373,6 +376,8 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,struct mlx5_core_dev *mdev = c->mdev;void *rqc = rqp->rqc;void *rqc_wq =MLX5_ADDR_OF (rqc, rqc, wq); +bool hd_split =MLX5E_HD_SPLIT (params) && (umem == (void *)0x1 ); +u32 num_xsk_frames =0 ;u32 rq_xdp_ix;u32 pool_size;int wq_sz;@@ -391,9 +396,10 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->mdev = mdev; rq->hw_mtu =MLX5E_SW2HW_MTU (params, params->sw_mtu); rq->xdpsq = &c->rq_xdpsq; - rq->umem = umem; +if (xsk) + rq->umem = umem; -if (rq->umem) +if (umem) rq->stats = &c->priv->channel_stats[c->ix].xskrq;else rq->stats = &c->priv->channel_stats[c->ix].rq;@@ -404,14 +410,18 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->xdp_prog = params->xdp_prog; rq_xdp_ix = rq->ix; -if (xsk) +if (umem) rq_xdp_ix += params->num_channels *MLX5E_RQ_GROUP_XSK ; err =xdp_rxq_info_reg (&rq->xdp_rxq, rq->netdev, rq_xdp_ix);if (err <0 ) goto err_rq_wq_destroy; +if (umem == (void *)0x1 ) + rq->buff.headroom =0 ; +else + rq->buff.headroom =mlx5e_get_rq_headroom (mdev, params, xsk); + rq->buff.map_dir = rq->xdp_prog ?DMA_BIDIRECTIONAL :DMA_FROM_DEVICE ; - rq->buff.headroom =mlx5e_get_rq_headroom (mdev, params, xsk); pool_size =1 << params->log_rq_mtu_frames;switch (rq->wq_type) {@@ -509,6 +519,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->wqe.skb_from_cqe = xsk ? mlx5e_xsk_skb_from_cqe_linear : + hd_split ? mlx5e_skb_from_cqe_nonlinear :mlx5e_rx_is_linear_skb (params,NULL ) ? mlx5e_skb_from_cqe_linear : mlx5e_skb_from_cqe_nonlinear;@@ -2035,13 +2046,19 @@ static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev,int frag_size_max =DEFAULT_FRAG_SIZE ;u32 buf_size =0 ;int i; +bool hd_split =MLX5E_HD_SPLIT (params) && xsk; + +if (hd_split) + frag_size_max =HD_SPLIT_DEFAULT_FRAG_SIZE ; +else + frag_size_max =DEFAULT_FRAG_SIZE ;#ifdef CONFIG_MLX5_EN_IPSEC if (MLX5_IPSEC_DEV (mdev)) byte_count +=MLX5E_METADATA_ETHER_LEN ;#endif -if (mlx5e_rx_is_linear_skb (params, xsk)) { +if (!hd_split &&mlx5e_rx_is_linear_skb (params, xsk)) {int frag_stride; frag_stride =mlx5e_rx_get_linear_frag_sz (params, xsk);@@ -2059,6 +2076,16 @@ static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev, frag_size_max =PAGE_SIZE ; i =0 ; + +if (hd_split) { +// Start with one fragment for all headers (implementing HDS) + info->arr[0 ].frag_size =TOTAL_HEADERS ; + info->arr[0 ].frag_stride =roundup_pow_of_two (PAGE_SIZE ); + buf_size +=TOTAL_HEADERS ; +// Now, continue with the payload frags. + i =1 ; + } +while (buf_size < byte_count) {int frag_size = byte_count - buf_size;@@ -2066,8 +2093,10 @@ static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev, frag_size =min (frag_size, frag_size_max); info->arr[i].frag_size = frag_size; - info->arr[i].frag_stride =roundup_pow_of_two (frag_size); - + info->arr[i].frag_stride =roundup_pow_of_two (hd_split ? +PAGE_SIZE : + frag_size); + info->arr[i].frag_source = hd_split; buf_size += frag_size; i++; }
Re: [RFC PATCH 06/21] mlx5: add header_split flag
Quoted 11 lines · Eric Dumazet
Quoted 4 lines · Jonathan Lemon
Adds a "
rx_hd_split" private flag parameter to ethtool.This enables header splitting, and sets up the fragment mappings. The feature is currently only enabled for netgpu channels.
We are using a similar idea (pseudo header split) to implement 4096+(headers) MTU at Google, to enable TCP RX zerocopy on x86.
Patch for mlx4 has not been sent upstream yet.
Quoted 7 lines
For mlx4, we are using a single buffer of 128*(number_of_slots_per_RX_RING), and 86 bytes for the first frag, so that the payload exactly fits a 4096 bytes page.
(In our case, most of our data TCP packets only have 12 bytes of TCP options)
I suggest that instead of a flag, you use a tunable, that can be set by ethtool, so that the exact number of bytes can be tuned, instead of hard coded in the driver.
I fully agree that such generic parameter would be a better solution than a private flag. But I have my doubts about adding more tunables. The point is that the concept of tunables looks like a workaround for the lack of extensibility of the ioctl interface where the space for adding new parameters to existing subcommands was limited (or none).
With netlink, adding new parameters is much easier and as only three tunables were added in 6 years (or four with your proposal), we don't have to worry about having too many different attributes (current code isn't even designed to scale well to many tunables).
This new header split parameter could IMHO be naturally put together with rx-copybreak and tx-copybreak and possibly any future parameters to control how packet contents is passed between NIC/driver and networking stack.
Quoted 2 lines
(Patch for the counter part of [1] was resent 10 days ago on netdev@ by Govindarajulu Varadarajan) (Not sure if this has been merged yet)
Not yet, I want to take another look in the rest of this week.
Michal
Quoted 16 lines
[1]
commit
f0db9b073415848709dd59a6394969882f517da9Author: Govindarajulu Varadarajan <[email protected]> Date: Wed Sep 3 03:17:20 2014 +0530ethtool: Add generic options for tunables
This patch adds new ethtool cmd,
ETHTOOL_GTUNABLEÐTOOL_STUNABLEfor getting tunable values from driver.Add
get_tunableandset_tunabletoethtool_ops. Driver implements these functions for getting/setting tunable value.Signed-off-by Govindarajulu Varadarajan <[email protected]>Signed-off-by David S. Miller <[email protected]>
Re: [RFC PATCH 06/21] mlx5: add header_split flag
Quoted 31 lines · Michal Kubecek
Quoted 11 lines · Eric Dumazet
Quoted 4 lines · Jonathan Lemon
Adds a "
rx_hd_split" private flag parameter to ethtool.This enables header splitting, and sets up the fragment mappings. The feature is currently only enabled for netgpu channels.
We are using a similar idea (pseudo header split) to implement 4096+(headers) MTU at Google, to enable TCP RX zerocopy on x86.
Patch for mlx4 has not been sent upstream yet.
Quoted 7 lines
For mlx4, we are using a single buffer of 128*(number_of_slots_per_RX_RING), and 86 bytes for the first frag, so that the payload exactly fits a 4096 bytes page.
(In our case, most of our data TCP packets only have 12 bytes of TCP options)
I suggest that instead of a flag, you use a tunable, that can be set by ethtool, so that the exact number of bytes can be tuned, instead of hard coded in the driver.
I fully agree that such generic parameter would be a better solution than a private flag. But I have my doubts about adding more tunables. The point is that the concept of tunables looks like a workaround for the lack of extensibility of the ioctl interface where the space for adding new parameters to existing subcommands was limited (or none).
With netlink, adding new parameters is much easier and as only three tunables were added in 6 years (or four with your proposal), we don't have to worry about having too many different attributes (current code isn't even designed to scale well to many tunables).
Quoted 4 lines
This new header split parameter could IMHO be naturally put together with rx-copybreak and tx-copybreak and possibly any future parameters to control how packet contents is passed between NIC/driver and networking stack.
This is what I suggested, maybe this was not clear.
Currently known tunables are :
enum tunable_id {
ETHTOOL_ID_UNSPEC,
ETHTOOL_RX_COPYBREAK,
ETHTOOL_TX_COPYBREAK,
ETHTOOL_PFC_PREVENTION_TOUT, /* timeout in msecs */
/*
* Add your fresh new tunable attribute above and remember to update
* tunable_strings[] in net/core/ethtool.c
*/
__ETHTOOL_TUNABLE_COUNT,
};
Ie add a new ETHTOOL_RX_HEADER_SPLIT value.
Or maybe I am misunderstanding your point.
Quoted 24 lines
Quoted 3 lines
(Patch for the counter part of [1] was resent 10 days ago on netdev@ by Govindarajulu Varadarajan) (Not sure if this has been merged yet)
Not yet, I want to take another look in the rest of this week.
Michal
Quoted 16 lines
[1]
commit
f0db9b073415848709dd59a6394969882f517da9Author: Govindarajulu Varadarajan <[email protected]> Date: Wed Sep 3 03:17:20 2014 +0530ethtool: Add generic options for tunables
This patch adds new ethtool cmd,
ETHTOOL_GTUNABLEÐTOOL_STUNABLEfor getting tunable values from driver.Add
get_tunableandset_tunabletoethtool_ops. Driver implements these functions for getting/setting tunable value.Signed-off-by Govindarajulu Varadarajan <[email protected]>Signed-off-by David S. Miller <[email protected]>
Re: [RFC PATCH 06/21] mlx5: add header_split flag
In reply to Eric DumazetQuoted 13 lines · Eric Dumazet
Quoted 4 lines · Jonathan Lemon
Adds a "
rx_hd_split" private flag parameter to ethtool.This enables header splitting, and sets up the fragment mappings. The feature is currently only enabled for netgpu channels.
We are using a similar idea (pseudo header split) to implement 4096+(headers) MTU at Google, to enable TCP RX zerocopy on x86.
Patch for mlx4 has not been sent upstream yet.
Quoted 8 lines
For mlx4, we are using a single buffer of 128*(number_of_slots_per_RX_RING), and 86 bytes for the first frag, so that the payload exactly fits a 4096 bytes page.
(In our case, most of our data TCP packets only have 12 bytes of TCP options)
I suggest that instead of a flag, you use a tunable, that can be set by ethtool, so that the exact number of bytes can be tuned, instead of hard coded in the driver.
Sounds reasonable - in the long run, it would be ideal to have the hardware actually perform header splitting, but for now using a tunable fixed offset will work. In the same vein, there should be a similar setting for the TCP option padding on the sender side.
Signature · 1 line
Jonathan
Re: [RFC PATCH 06/21] mlx5: add header_split flag
Quoted 20 lines · Jonathan Lemon
Quoted 19 lines · Eric Dumazet
Quoted 4 lines · Jonathan Lemon
Adds a "
rx_hd_split" private flag parameter to ethtool.This enables header splitting, and sets up the fragment mappings. The feature is currently only enabled for netgpu channels.
We are using a similar idea (pseudo header split) to implement 4096+(headers) MTU at Google, to enable TCP RX zerocopy on x86.
Patch for mlx4 has not been sent upstream yet.
For mlx4, we are using a single buffer of 128*(number_of_slots_per_RX_RING), and 86 bytes for the first frag, so that the payload exactly fits a 4096 bytes page.
(In our case, most of our data TCP packets only have 12 bytes of TCP options)
Quoted 8 lines
Quoted 2 lines
I suggest that instead of a flag, you use a tunable, that can be set by ethtool, so that the exact number of bytes can be tuned, instead of hard coded in the driver.
Sounds reasonable - in the long run, it would be ideal to have the hardware actually perform header splitting, but for now using a tunable fixed offset will work. In the same vein, there should be a similar setting for the TCP option padding on the sender side.
Some NIC have variable header split (Intel ixgbe I am pretty sure)
We use a mix of NIC, some with variable header splits, some with fixed pseudo header split (mlx4)
Because of this, we had to limit TCP advmss to 4108 (4096 + 12), regardless of the NIC abilities.
Re: [RFC PATCH 06/21] mlx5: add header_split flag
In reply to Jonathan LemonQuoted 2 lines · Jonathan Lemon
In the same vein, there should be a similar setting for the TCP option padding on the sender side.
We had no need for such hack in the TCP stack :)
kernel test robot
build botRe: [RFC PATCH 06/21] mlx5: add header_split flag
In reply to Jonathan Lemon[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on net/master]
[also build test WARNING on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: sparc-allyesconfig (attached as .config) compiler: sparc64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=sparc
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>, old ones prefixed by <<):
drivers/net/ethernet/mellanox/mlx5/core/en_main.c: In function 'mlx5e_alloc_rq':drivers/net/ethernet/mellanox/mlx5/core/en_main.c:380:6: warning: unused variable 'num_xsk_frames' [-Wunused-variable] 380 | u32 num_xsk_frames = 0;
| ^~~~~~~~~~~~~~
In file included from include/linux/kernel.h:13,
from include/asm-generic/bug.h:19,
from arch/sparc/include/asm/bug.h:25,
from include/linux/bug.h:5,
from include/linux/refcount.h:96,
from include/net/act_api.h:9,
from include/net/tc_act/tc_gact.h:5,
from drivers/net/ethernet/mellanox/mlx5/core/en_main.c:33:drivers/net/ethernet/mellanox/mlx5/core/en_main.c: In function 'mlx5e_build_rq_frags_info':
<< from drivers/net/ethernet/mellanox/mlx5/core/en_main.c:42:
>> drivers/net/ethernet/mellanox/mlx5/core/en_main.c:2097:24: warning: ?: using integer constants in boolean context [-Wint-in-bool-context]
2097 | PAGE_SIZE :include/linux/log2.h:176:4: note: in definition of macro 'roundup_pow_of_two'
176 | (n == 1) ? 1 : | ^
vim +2097 drivers/net/ethernet/mellanox/mlx5/core/en_main.c
2060
2061 if (!hd_split && mlx5e_rx_is_linear_skb(params, xsk)) {
2062 int frag_stride;
2063
2064 frag_stride = mlx5e_rx_get_linear_frag_sz(params, xsk);
2065 frag_stride = roundup_pow_of_two(frag_stride);
2066
2067 info->arr[0].frag_size = byte_count;
2068 info->arr[0].frag_stride = frag_stride;
2069 info->num_frags = 1;
2070 info->wqe_bulk = PAGE_SIZE / frag_stride;
2071 goto out;
2072 }
2073
2074 if (byte_count > PAGE_SIZE +
2075 (MLX5E_MAX_RX_FRAGS - 1) * frag_size_max)
2076 frag_size_max = PAGE_SIZE;
2077
2078 i = 0;
2079
2080 if (hd_split) {
2081 // Start with one fragment for all headers (implementing HDS)
2082 info->arr[0].frag_size = TOTAL_HEADERS;
2083 info->arr[0].frag_stride = roundup_pow_of_two(PAGE_SIZE);
2084 buf_size += TOTAL_HEADERS;
2085 // Now, continue with the payload frags.
2086 i = 1;
2087 }
2088
2089 while (buf_size < byte_count) {
2090 int frag_size = byte_count - buf_size;
2091
2092 if (i < MLX5E_MAX_RX_FRAGS - 1)
2093 frag_size = min(frag_size, frag_size_max);
2094
2095 info->arr[i].frag_size = frag_size;
2096 info->arr[i].frag_stride = roundup_pow_of_two(hd_split ?
> 2097 PAGE_SIZE :
2098 frag_size);
2099 info->arr[i].frag_source = hd_split;
2100 buf_size += frag_size;
2101 i++;
2102 }
2103 info->num_frags = i;
2104 /* number of different wqes sharing a page */
2105 info->wqe_bulk = 1 + (info->num_frags % 2);
2106
2107 out:
2108 info->wqe_bulk = max_t(u8, info->wqe_bulk, 8);
2109 info->log_num_frags = order_base_2(info->num_frags);
2110 }
2111
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 7/21 mlx5: remove the umem parameter from mlx5e_open_channel
In reply to Jonathan LemonInstead of obtaining the umem parameter from the channel parameters and passing it to the function, push this down into the function itself.
Move xsk open logic into its own function, in preparation for the upcoming netgpu commit.
--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 35 +++++++++++++------ 1 file changed, 24 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +24 -11
@@ -1935,15 +1935,33 @@ static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)return (ix + port_aff_bias) %mlx5e_get_num_lag_ports (mdev); } +static int +mlx5e_xsk_optional_open (struct mlx5e_priv *priv,int ix, +struct mlx5e_params *params, +struct mlx5e_channel_param *cparam, +struct mlx5e_channel *c) +{ +struct mlx5e_xsk_param xsk; +struct xdp_umem *umem; +int err =0 ; + + umem =mlx5e_xsk_get_umem (params, params->xsk, ix); + +if (umem) { +mlx5e_build_xsk_param (umem, &xsk); + err =mlx5e_open_xsk (priv, params, &xsk, umem, c); + } + +return err; +} +static int mlx5e_open_channel (struct mlx5e_priv *priv,int ix,struct mlx5e_params *params,struct mlx5e_channel_param *cparam, -struct xdp_umem *umem,struct mlx5e_channel **cp) {int cpu =cpumask_first (mlx5_comp_irq_get_affinity_mask (priv->mdev, ix));struct net_device *netdev = priv->netdev; -struct mlx5e_xsk_param xsk;struct mlx5e_channel *c;unsigned int irq;int err;
@@ -1977,9 +1995,9 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,if (unlikely (err)) goto err_napi_del; -if (umem) { -mlx5e_build_xsk_param (umem, &xsk); - err =mlx5e_open_xsk (priv, params, &xsk, umem, c); +/* This opens a second set of shadow queues for xsk */ +if (params->xdp_prog) { + err =mlx5e_xsk_optional_open (priv, ix, params, cparam, c);if (unlikely (err)) goto err_close_queues; }
@@ -2345,12 +2363,7 @@ int mlx5e_open_channels(struct mlx5e_priv *priv,mlx5e_build_channel_param (priv, &chs->params, cparam);for (i =0 ; i < chs->num; i++) { -struct xdp_umem *umem =NULL ; - -if (chs->params.xdp_prog) - umem =mlx5e_xsk_get_umem (&chs->params, chs->params.xsk, i); - - err =mlx5e_open_channel (priv, i, &chs->params, cparam, umem, &chs->c[i]); + err =mlx5e_open_channel (priv, i, &chs->params, cparam, &chs->c[i]);if (err) goto err_close_channels; }
Signature · 1 line
2.24.1
RFC PATCH 8/21 misc: add shqueue.h for prototyping
In reply to Jonathan LemonShared queues between user and kernel use their own private structures for accessing a shared data area, but they need to use the same queue functions.
Rather than doing the 'right' thing and duplicating the file for each domain, temporary cheat for prototyping and use a single shared file.
--- include/uapi/misc/shqueue.h | 205 ++++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 include/uapi/misc/shqueue.h
include/uapi/misc/shqueue.h new file +205 -0
@@ -0,0 +1,205 @@ +#pragma once + +/* XXX +* This is not a user api, but placed here for prototyping, in order to +* avoid two nigh identical copies for user and kernel space. +*/ + +/* kernel only */ +struct shared_queue_map { +unsigned prod____cacheline_aligned_in_smp ; +unsigned cons____cacheline_aligned_in_smp ; +char data[]____cacheline_aligned_in_smp ; +}; + +/* user and kernel private copy - identical in order to share sq* fcns */ +struct shared_queue { +unsigned *prod; +unsigned *cons; +char *data; +unsigned elt_sz; +unsigned mask; +unsigned cached_prod; +unsigned cached_cons; +unsigned entries; + +unsigned map_sz; +void *map_ptr; +}; + +/* +* see documenation in tools/include/linux/ring_buffer.h +* using explicit smp_/_ONCE is an optimization over smp_{store|load} +*/ + +static inline void __sq_load_acquire_cons (struct shared_queue *q) +{ +/* Refresh the local tail pointer */ + q->cached_cons =READ_ONCE (*q->cons); +/* A, matches D */ +} + +static inline void __sq_store_release_cons (struct shared_queue *q) +{ +smp_mb ();/* D, matches A */ +WRITE_ONCE (*q->cons, q->cached_cons); +} + +static inline void __sq_load_acquire_prod (struct shared_queue *q) +{ +/* Refresh the local pointer */ + q->cached_prod =READ_ONCE (*q->prod); +smp_rmb ();/* C, matches B */ +} + +static inline void __sq_store_release_prod (struct shared_queue *q) +{ +smp_wmb ();/* B, matches C */ +WRITE_ONCE (*q->prod, q->cached_prod); +} + +static inline void sq_cons_refresh (struct shared_queue *q) +{ +__sq_store_release_cons (q); +__sq_load_acquire_prod (q); +} + +static inline bool sq_empty (struct shared_queue *q) +{ +return READ_ONCE (*q->prod) ==READ_ONCE (*q->cons); +} + +static inline bool sq_cons_empty (struct shared_queue *q) +{ +return q->cached_prod == q->cached_cons; +} + +static inline unsigned __sq_cons_ready (struct shared_queue *q) +{ +return q->cached_prod - q->cached_cons; +} + +static inline unsigned sq_cons_ready (struct shared_queue *q) +{ +if (q->cached_prod == q->cached_cons) +__sq_load_acquire_prod (q); + +return q->cached_prod - q->cached_cons; +} + +static inline bool sq_cons_avail (struct shared_queue *q,unsigned count) +{ +if (count <=__sq_cons_ready (q)) +return true; +__sq_load_acquire_prod (q); +return count <=__sq_cons_ready (q); +} + +static inline void *sq_get_ptr (struct shared_queue *q,unsigned idx) +{ +return q->data + (idx & q->mask) * q->elt_sz; +} + +static inline void sq_cons_complete (struct shared_queue *q) +{ +__sq_store_release_cons (q); +} + +static inline void *sq_cons_peek (struct shared_queue *q) +{ +if (sq_cons_empty (q)) { +sq_cons_refresh (q); +if (sq_cons_empty (q)) +return NULL ; + } +return sq_get_ptr (q, q->cached_cons); +} + +static inline unsigned +sq_peek_batch (struct shared_queue *q,void **ptr,unsigned count) +{ +unsigned i, idx, ready; + + ready =sq_cons_ready (q); +if (!ready) +return 0 ; + + count = count > ready ? ready : count; + + idx = q->cached_cons; +for (i =0 ; i < count; i++) + ptr[i] =sq_get_ptr (q, idx++); + + q->cached_cons += count; + +return count; +} + +static inline unsigned +sq_cons_batch (struct shared_queue *q,void **ptr,unsigned count) +{ +unsigned i, idx, ready; + + ready =sq_cons_ready (q); +if (!ready) +return 0 ; + + count = count > ready ? ready : count; + + idx = q->cached_cons; +for (i =0 ; i < count; i++) + ptr[i] =sq_get_ptr (q, idx++); + + q->cached_cons += count; +sq_cons_complete (q); + +return count; +} + +static inline void sq_cons_advance (struct shared_queue *q) +{ + q->cached_cons++; +} + +static inline unsigned __sq_prod_space (struct shared_queue *q) +{ +return q->entries - (q->cached_prod - q->cached_cons); +} + +static inline unsigned sq_prod_space (struct shared_queue *q) +{ +unsigned space; + + space =__sq_prod_space (q); +if (!space) { +__sq_load_acquire_cons (q); + space =__sq_prod_space (q); + } +return space; +} + +static inline bool sq_prod_avail (struct shared_queue *q,unsigned count) +{ +if (count <=__sq_prod_space (q)) +return true; +__sq_load_acquire_cons (q); +return count <=__sq_prod_space (q); +} + +static inline void *sq_prod_get_ptr (struct shared_queue *q) +{ +return sq_get_ptr (q, q->cached_prod++); +} + +static inline void *sq_prod_reserve (struct shared_queue *q) +{ +if (!sq_prod_space (q)) +return NULL ; + +return sq_prod_get_ptr (q); +} + +static inline void sq_prod_submit (struct shared_queue *q) +{ +__sq_store_release_prod (q); +}
Signature · 1 line
2.24.1
kernel test robot
build botRe: [RFC PATCH 08/21] misc: add shqueue.h for prototyping
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net/master]
[also build test ERROR on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: x86_64-randconfig-s021-20200618 (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.2-rc1-10-gc17b1b06-dirty # save the attached .config to linux build tree make W=1 C=1 ARCH=x86_64 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__'
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from <command-line>:32:
>> ./usr/include/misc/shqueue.h:10:16: error: expected ':', ',', ';', '}' or '__attribute__' before '____cacheline_aligned_in_smp'
10 | unsigned prod ____cacheline_aligned_in_smp;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
./usr/include/misc/shqueue.h: In function '__sq_load_acquire_cons':
>> ./usr/include/misc/shqueue.h:38:19: error: implicit declaration of function 'READ_ONCE' [-Werror=implicit-function-declaration]
38 | q->cached_cons = READ_ONCE(*q->cons);
| ^~~~~~~~~
./usr/include/misc/shqueue.h: In function '__sq_store_release_cons':
>> ./usr/include/misc/shqueue.h:44:2: error: implicit declaration of function 'smp_mb' [-Werror=implicit-function-declaration]
44 | smp_mb(); /* D, matches A */
| ^~~~~~
>> ./usr/include/misc/shqueue.h:45:2: error: implicit declaration of function 'WRITE_ONCE' [-Werror=implicit-function-declaration]
45 | WRITE_ONCE(*q->cons, q->cached_cons);
| ^~~~~~~~~~
./usr/include/misc/shqueue.h: In function '__sq_load_acquire_prod':
>> ./usr/include/misc/shqueue.h:52:2: error: implicit declaration of function 'smp_rmb' [-Werror=implicit-function-declaration]
52 | smp_rmb(); /* C, matches B */
| ^~~~~~~
./usr/include/misc/shqueue.h: In function '__sq_store_release_prod':
>> ./usr/include/misc/shqueue.h:57:2: error: implicit declaration of function 'smp_wmb' [-Werror=implicit-function-declaration]
57 | smp_wmb(); /* B, matches C */
| ^~~~~~~
./usr/include/misc/shqueue.h: At top level:
>> ./usr/include/misc/shqueue.h:67:19: error: unknown type name 'bool'
67 | static __inline__ bool sq_empty(struct shared_queue *q)
| ^~~~
./usr/include/misc/shqueue.h:72:19: error: unknown type name 'bool'
72 | static __inline__ bool sq_cons_empty(struct shared_queue *q)
| ^~~~
./usr/include/misc/shqueue.h:90:19: error: unknown type name 'bool'
90 | static __inline__ bool sq_cons_avail(struct shared_queue *q, unsigned count)
| ^~~~
./usr/include/misc/shqueue.h: In function 'sq_cons_avail':
>> ./usr/include/misc/shqueue.h:93:10: error: 'true' undeclared (first use in this function)
93 | return true;
| ^~~~
./usr/include/misc/shqueue.h:93:10: note: each undeclared identifier is reported only once for each function it appears in
./usr/include/misc/shqueue.h: In function 'sq_cons_peek':
>> ./usr/include/misc/shqueue.h:113:11: error: 'NULL' undeclared (first use in this function)
113 | return NULL;
| ^~~~
./usr/include/misc/shqueue.h:1:1: note: 'NULL' is defined in header '<stddef.h>'; did you forget to '#include <stddef.h>'?
+++ |+#include <stddef.h>
1 | #pragma once
./usr/include/misc/shqueue.h: At top level:
./usr/include/misc/shqueue.h:181:19: error: unknown type name 'bool'
181 | static __inline__ bool sq_prod_avail(struct shared_queue *q, unsigned count)
| ^~~~
./usr/include/misc/shqueue.h: In function 'sq_prod_avail':
./usr/include/misc/shqueue.h:184:10: error: 'true' undeclared (first use in this function)
184 | return true;
| ^~~~
./usr/include/misc/shqueue.h: In function 'sq_prod_reserve':
./usr/include/misc/shqueue.h:197:10: error: 'NULL' undeclared (first use in this function)
197 | return NULL;
| ^~~~ ./usr/include/misc/shqueue.h:197:10: note: 'NULL' is defined in header '<stddef.h>'; did you forget to '#include <stddef.h>'?
cc1: some warnings being treated as errors
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 9/21 include: add definitions for netgpu
In reply to Jonathan LemonThis adds the netgpu structure (which arguably should be private), and some cruft to support using netgpu as a loadable module, which should disappear.
--- include/net/netgpu.h | 65 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 include/net/netgpu.h
include/net/netgpu.h new file +65 -0
@@ -0,0 +1,65 @@ +#pragma once + +struct net_device ; +#include <uapi/misc/shqueue.h> /* XXX */ + +struct netgpu_pgcache { +struct netgpu_pgcache *next; +struct page *page[]; +}; + +struct netgpu_ctx { +struct xarray xa;/* contains regions */ +unsigned int index; +refcount_t ref; +struct shared_queue fill; +struct shared_queue rx; +struct net_device *dev; +struct netgpu_pgcache *napi_cache; +struct netgpu_pgcache *spare_cache; +struct netgpu_pgcache *any_cache; +spinlock_t pgcache_lock; +struct page *dummy_page; +unsigned page_extra_refc; +int queue_id; +int napi_cache_count; +int any_cache_count; +struct user_struct *user; +unsigned account_mem :1 ; +}; + +int netgpu_get_page (struct netgpu_ctx *ctx,struct page **page, +dma_addr_t *dma); +void netgpu_put_page (struct netgpu_ctx *ctx,struct page *page,bool napi); +int netgpu_get_pages (struct sock *sk,struct page **pages,unsigned long addr, +int count); + +/*---------------------------------------------------------------------------*/ +/* XXX temporary development support */ + +extern int (*fn_netgpu_get_page)(struct netgpu_ctx *ctx, +struct page **page,dma_addr_t *dma); +extern void (*fn_netgpu_put_page)(struct netgpu_ctx *,struct page *,bool ); +extern int (*fn_netgpu_get_pages)(struct sock *,struct page **, +unsigned long ,int ); +extern struct netgpu_ctx *g_ctx; + +static inline int +__netgpu_get_page (struct netgpu_ctx *ctx, +struct page **page,dma_addr_t *dma) +{ +return fn_netgpu_get_page (ctx, page, dma); +} + +static inline void +__netgpu_put_page (struct netgpu_ctx *ctx,struct page *page,bool napi) +{ +return fn_netgpu_put_page (ctx, page, napi); +} + +static inline int +__netgpu_get_pages (struct sock *sk,struct page **pages, +unsigned long addr,int count) +{ +return fn_netgpu_get_pages (sk, pages, addr, count); +}
Signature · 1 line
2.24.1
RFC PATCH 10/21 mlx5: add netgpu queue functions
In reply to Jonathan LemonAdd the netgpu setup/teardown functions, which are not hooked up yet. The driver also handles netgpu module loading and unloading.
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 3 +- .../mellanox/mlx5/core/en/netgpu/setup.c | 475 ++++++++++++++++++ .../mellanox/mlx5/core/en/netgpu/setup.h | 42 ++ 3 files changed, 519 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +2 -1
@@ -25,7 +25,8 @@ mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \ en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \ en_selftest.o en/port.o en/monitor_stats.o en/health.o \ en/reporter_tx.o en/reporter_rx.o en/params.o en/xsk/umem.o \ - en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o en/devlink.o + en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o en/devlink.o \ + en/netgpu/setup.o # # Netdev extra
drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c new file +475 -0
475 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.h new file +42 -0
42 lines not on this page · raw message
Signature · 1 line
2.24.1
RFC PATCH 11/21 skbuff: add a zc_netgpu bitflag
In reply to Jonathan LemonThis could likely be moved elsewhere. The presence of the flag on the skb indicates that one of the fragments may contain zerocopy data (where the data is not accessible to the cpu).
--- include/linux/skbuff.h | 3 ++- net/core/skbuff.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/skbuff.h +2 -1
@@ -782,7 +782,8 @@ struct sk_buff { fclone:2, peeked:1, head_frag:1, - pfmemalloc:1; + pfmemalloc:1, + zc_netgpu:1;#ifdef CONFIG_SKB_EXTENSIONS __u8 active_extensions;#endif
net/core/skbuff.c +1 -0
@@ -992,6 +992,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb) n->cloned =1 ; n->nohdr =0 ; n->peeked =0 ; +C (zc_netgpu);C (pfmemalloc); n->destructor =NULL ;C (tail);
Signature · 1 line
2.24.1
RFC PATCH 12/21 mlx5: hook up the netgpu channel functions
In reply to Jonathan LemonHook up all the netgpu plumbing, except the enable/disable calls. Those will be added after the netgpu module itself.
--- .../mellanox/mlx5/core/en/netgpu/setup.c | 2 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 35 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/en_rx.c | 52 +++++++++++++++++-- .../net/ethernet/mellanox/mlx5/core/en_txrx.c | 15 +++++- 4 files changed, 97 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/netgpu/setup.c +1 -1
@@ -78,7 +78,7 @@ mlx5e_netgpu_avail(struct mlx5e_rq *rq, u8 count)* doesn't consider any_cache_count. */ return ctx->napi_cache_count >= count || -sq_cons_ready (&ctx->fill) >= (count - ctx->napi_cache_count); +sq_cons_avail (&ctx->fill, count - ctx->napi_cache_count); }void mlx5e_netgpu_taken (struct mlx5e_rq *rq)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +35 -0
@@ -62,6 +62,7 @@#include "en/xsk/setup.h" #include "en/xsk/rx.h" #include "en/xsk/tx.h" +#include "en/netgpu/setup.h" #include "en/hv_vhca_stats.h" #include "en/devlink.h" #include "lib/mlx5.h"
@@ -1955,6 +1956,24 @@ mlx5e_xsk_optional_open(struct mlx5e_priv *priv, int ix,return err; } +static int +mlx5e_netgpu_optional_open (struct mlx5e_priv *priv,int ix, +struct mlx5e_params *params, +struct mlx5e_channel_param *cparam, +struct mlx5e_channel *c) +{ +struct netgpu_ctx *ctx; +int err =0 ; + + ctx =mlx5e_netgpu_get_ctx (params, params->xsk, ix); + +if (ctx) + err =mlx5e_open_netgpu (priv, params, ctx, c); + +return err; +} + + static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, struct mlx5e_params *params, struct mlx5e_channel_param *cparam,
@@ -2002,6 +2021,13 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, goto err_close_queues; } +/* This opens a second set of shadow queues for netgpu */ +if (params->hd_split) { + err =mlx5e_netgpu_optional_open (priv, ix, params, cparam, c); +if (unlikely (err)) + goto err_close_queues; + } + *cp = c;return 0 ;
@@ -2037,6 +2063,9 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)if (test_bit (MLX5E_CHANNEL_STATE_XSK , c->state))mlx5e_deactivate_xsk (c); +if (test_bit (MLX5E_CHANNEL_STATE_NETGPU , c->state)) +mlx5e_deactivate_netgpu (c); +mlx5e_deactivate_rq (&c->rq);mlx5e_deactivate_icosq (&c->icosq);for (tc =0 ; tc < c->num_tc; tc++)
@@ -2047,6 +2076,10 @@ static void mlx5e_close_channel(struct mlx5e_channel *c) {if (test_bit (MLX5E_CHANNEL_STATE_XSK , c->state))mlx5e_close_xsk (c); + +if (test_bit (MLX5E_CHANNEL_STATE_NETGPU , c->state)) +mlx5e_close_netgpu (c); +mlx5e_close_queues (c);netif_napi_del (&c->napi);
@@ -3012,11 +3045,13 @@ void mlx5e_activate_priv_channels(struct mlx5e_priv *priv)mlx5e_redirect_rqts_to_channels (priv, &priv->channels);mlx5e_xsk_redirect_rqts_to_channels (priv, &priv->channels); +mlx5e_netgpu_redirect_rqts_to_channels (priv, &priv->channels); }void mlx5e_deactivate_priv_channels (struct mlx5e_priv *priv) {mlx5e_xsk_redirect_rqts_to_drop (priv, &priv->channels); +mlx5e_netgpu_redirect_rqts_to_drop (priv, &priv->channels);mlx5e_redirect_rqts_to_drop (priv);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +47 -5
113 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +14 -1
@@ -122,6 +122,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)struct mlx5e_rq *xskrq = &c->xskrq;struct mlx5e_rq *rq = &c->rq;bool xsk_open =test_bit (MLX5E_CHANNEL_STATE_XSK , c->state); +bool netgpu_open =test_bit (MLX5E_CHANNEL_STATE_NETGPU , c->state);bool aff_change = false;bool busy_xsk = false;bool busy = false;
@@ -139,7 +140,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) busy |=mlx5e_poll_xdpsq_cq (&c->rq_xdpsq.cq);if (likely (budget)) {/* budget=0 means: don't poll rx rings */ -if (xsk_open) +if (xsk_open || netgpu_open) work_done =mlx5e_poll_rx_cq (&xskrq->cq, budget);if (likely (budget - work_done))
@@ -154,6 +155,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) mlx5e_post_rx_mpwqes, mlx5e_post_rx_wqes, rq); + +if (netgpu_open) { +mlx5e_poll_ico_cq (&c->xskicosq.cq); + busy_xsk |= xskrq->post_wqes (xskrq); + } +if (xsk_open) {if (mlx5e_poll_ico_cq (&c->xskicosq.cq))/* Don't clear the flag if nothing was polled to prevent
@@ -191,6 +198,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)mlx5e_cq_arm (&c->icosq.cq);mlx5e_cq_arm (&c->xdpsq.cq); +if (netgpu_open) { +mlx5e_handle_rx_dim (xskrq); +mlx5e_cq_arm (&c->xskicosq.cq); +mlx5e_cq_arm (&xskrq->cq); + } +if (xsk_open) {mlx5e_handle_rx_dim (xskrq);mlx5e_cq_arm (&c->xskicosq.cq);
Signature · 1 line
2.24.1
RFC PATCH 13/21 netdevice: add SETUP_NETGPU to the netdev_bpf structure
In reply to Jonathan LemonThis command will be used to setup/tear down netgpu queues.
--- include/linux/netdevice.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/netdevice.h +6 -0
@@ -880,6 +880,7 @@ enum bpf_netdev_command {BPF_OFFLOAD_MAP_ALLOC ,BPF_OFFLOAD_MAP_FREE ,XDP_SETUP_XSK_UMEM , + XDP_SETUP_NETGPU, };struct bpf_prog_offload_ops ;
@@ -911,6 +912,11 @@ struct netdev_bpf {struct xdp_umem *umem;u16 queue_id; } xsk; +/* XDP_SETUP_NETGPU */ +struct { +struct netgpu_ctx *ctx; +u16 queue_id; + } netgpu; }; };
Signature · 1 line
2.24.1
RFC PATCH 14/21 kernel: export free_uid
In reply to Jonathan Lemonget_uid is a static inline which can be called from a module, so free_uid should also be callable.
--- kernel/user.c | 1 + 1 file changed, 1 insertion(+)
kernel/user.c +1 -0
@@ -171,6 +171,7 @@ void free_uid(struct user_struct *up)if (refcount_dec_and_lock_irqsave (&up->__count, &uidhash_lock, &flags))free_user (up, flags); } +EXPORT_SYMBOL_GPL (free_uid);struct user_struct *alloc_uid (kuid_t uid) {
Signature · 1 line
2.24.1
RFC PATCH 15/21 netgpu: add network/gpu dma module
In reply to Jonathan LemonNetgpu provides a data path for zero-copy TCP sends and receives directly to GPU memory. TCP processing is done on the host CPU, while data is DMA'd to and from device memory.
The use case for this module are GPUs used for machine learning, which are located near the NICs, and have a high bandwith PCI connection between the GPU/NIC.
This initial working code is a proof of concept, for discussion.
--- drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/netgpu/Kconfig | 10 + drivers/misc/netgpu/Makefile | 11 + drivers/misc/netgpu/nvidia.c | 1516 ++++++++++++++++++++++++++++++++++ include/uapi/misc/netgpu.h | 43 + 6 files changed, 1582 insertions(+) create mode 100644 drivers/misc/netgpu/Kconfig create mode 100644 drivers/misc/netgpu/Makefile create mode 100644 drivers/misc/netgpu/nvidia.c create mode 100644 include/uapi/misc/netgpu.h
drivers/misc/Kconfig +1 -0
@@ -472,4 +472,5 @@ source "drivers/misc/ocxl/Kconfig" source "drivers/misc/cardreader/Kconfig" source "drivers/misc/habanalabs/Kconfig" source "drivers/misc/uacce/Kconfig" +source "drivers/misc/netgpu/Kconfig" endmenu
drivers/misc/Makefile +1 -0
@@ -57,3 +57,4 @@ obj-$(CONFIG_PVPANIC) += pvpanic.o obj-$(CONFIG_HABANA_AI) += habanalabs/ obj-$(CONFIG_UACCE) += uacce/ obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o +obj-$(CONFIG_NETGPU) += netgpu/
drivers/misc/netgpu/Kconfig new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# NetGPU framework +# + +config NETGPU + tristate "Network/GPU driver" + depends on PCI + ---help--- + Experimental Network / GPU driver
drivers/misc/netgpu/Makefile new file +11 -0
@@ -0,0 +1,11 @@ +# SPDX-License-Identifier: GPL-2.0-only + +pkg = /home/bsd/local/pull/nvidia/NVIDIA-Linux-x86_64-440.59/kernel + +obj-$(CONFIG_NETGPU) := netgpu.o + +netgpu-y := nvidia.o + +# netgpu-$(CONFIG_DEBUG_FS) += debugfs.o + +ccflags-y += -I$(pkg)
drivers/misc/netgpu/nvidia.c new file +1516 -0
1,516 lines not on this page · raw message
include/uapi/misc/netgpu.h new file +43 -0
43 lines not on this page · raw message
Signature · 1 line
2.24.1
RFC PATCH 16/21 lib: have __zerocopy_sg_from_iter get netgpu pages for a sk
In reply to Jonathan LemonIf a sock is marked as sending zc data, have the iterator retrieve the correct zc pages from the netgpu module.
--- include/linux/uio.h | 4 ++++ lib/iov_iter.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ net/core/datagram.c | 6 +++++- 3 files changed, 54 insertions(+), 1 deletion(-)
include/linux/uio.h +4 -0
@@ -227,6 +227,10 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages,ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages); +struct sock ; +ssize_t iov_iter_sk_get_pages (struct iov_iter *i,struct sock *sk, +size_t maxsize,struct page **pages,unsigned maxpages, +size_t *pgoff);const void *dup_iter (struct iov_iter *new,struct iov_iter *old,gfp_t flags);
lib/iov_iter.c +45 -0
57 lines not on this page · raw message
net/core/datagram.c +5 -1
@@ -530,6 +530,10 @@ int skb_copy_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len) {trace_skb_copy_datagram_iovec (skb ,len ); +if (skb->zc_netgpu) { +pr_err ("skb netgpu datagram on !netgpu sk\n" ); +return -EFAULT ; + }return __skb_datagram_iter (skb, offset, to, len, false, simple_copy_to_iter,NULL ); }
@@ -631,7 +635,7 @@ int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb,if (frag ==MAX_SKB_FRAGS )return -EMSGSIZE ; - copied =iov_iter_get_pages (from, pages, length, + copied =iov_iter_sk_get_pages (from, sk, length, pages,MAX_SKB_FRAGS - frag, &start);if (copied <0 )return -EFAULT ;
Signature · 1 line
2.24.1
kernel test robot
build botRe: [RFC PATCH 16/21] lib: have __zerocopy_sg_from_iter get netgpu pages for a sk
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net/master]
[also build test ERROR on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master ipvs/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: s390-allnoconfig (attached as .config) compiler: s390-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=s390
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
s390-linux-ld: lib/iov_iter.o: in function `iov_iter_sk_get_pages':
iov_iter.c:(.text+0x33fa): undefined reference to `fn_netgpu_get_pages'
>> s390-linux-ld: iov_iter.c:(.text+0x34ae): undefined reference to `fn_netgpu_get_pages'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 16/21] lib: have __zerocopy_sg_from_iter get netgpu pages for a sk
In reply to Jonathan Lemon[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net/master]
[also build test ERROR on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: x86_64-kexec (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
ld: lib/iov_iter.o: in function `__netgpu_get_pages':
>> include/net/netgpu.h:64: undefined reference to `fn_netgpu_get_pages' >> ld: include/net/netgpu.h:64: undefined reference to `fn_netgpu_get_pages'
vim +64 include/net/netgpu.h
60654acea6f2ba Jonathan Lemon 2020-06-18 59
60654acea6f2ba Jonathan Lemon 2020-06-18 60 static inline int
60654acea6f2ba Jonathan Lemon 2020-06-18 61 __netgpu_get_pages(struct sock *sk, struct page **pages,
60654acea6f2ba Jonathan Lemon 2020-06-18 62 unsigned long addr, int count)
60654acea6f2ba Jonathan Lemon 2020-06-18 63 {60654acea6f2ba Jonathan Lemon 2020-06-18 @64 return fn_netgpu_get_pages(sk, pages, addr, count);
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 17/21 net/core: add the SO_REGISTER_DMA socket option
In reply to Jonathan LemonThis option says that the socket will be performing zero copy sends and receives through the netgpu module.
--- include/uapi/asm-generic/socket.h | 2 ++ net/core/sock.c | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+)
include/uapi/asm-generic/socket.h +2 -0
@@ -119,6 +119,8 @@#define SO_DETACH_REUSEPORT_BPF 68 +#define SO_REGISTER_DMA 69 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64 || (defined(__x86_64__) && defined(__ILP32__))
net/core/sock.c +26 -0
@@ -828,6 +828,25 @@ void sock_set_rcvbuf(struct sock *sk, int val) }EXPORT_SYMBOL (sock_set_rcvbuf); +extern int netgpu_register_dma (struct sock *sk,char __user *optval,unsigned int optlen); + +static int +sock_register_dma (struct sock *sk,char __user *optval,unsigned int optlen) +{ +int rc; +int (*fn)(struct sock *sk,char __user *optval,unsigned int optlen); + + fn =symbol_get (netgpu_register_dma); +if (!fn) +return -EINVAL ; + + rc =fn (sk, optval, optlen); + +symbol_put (netgpu_register_dma); + +return rc; +} +/* * This is meant for all protocols to use and covers goings on * at the socket level. Everything here is generic.
@@ -1232,6 +1251,13 @@ int sock_setsockopt(struct socket *sock, int level, int optname, }break ; +case SO_REGISTER_DMA : +if (!sk->sk_bound_dev_if) + ret = -EINVAL ; +else + ret =sock_register_dma (sk, optval, optlen); +break ; +case SO_TXTIME :if (optlen !=sizeof (struct sock_txtime )) { ret = -EINVAL ;
Signature · 1 line
2.24.1
RFC PATCH 18/21 tcp: add MSG_NETDMA flag for sendmsg()
In reply to Jonathan LemonThis flag indicates that the attached data is a zero-copy send, and the pages should be retrieved from the netgpu module. The socket must have been marked as SOCK_ZEROCOPY, and also registered with netgpu via SO_REGISTER_DMA.
--- include/linux/socket.h | 1 + net/ipv4/tcp.c | 8 ++++++++ 2 files changed, 9 insertions(+)
include/linux/socket.h +1 -0
@@ -310,6 +310,7 @@ struct ucred {*/ #define MSG_ZEROCOPY 0x4000000/* Use user data in kernel path */ +#define MSG_NETDMA 0x8000000#define MSG_FASTOPEN 0x20000000/* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000/* Set close_on_exec for file descriptor received through
net/ipv4/tcp.c +8 -0
@@ -1209,6 +1209,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) uarg->zerocopy =0 ; } +if (flags &MSG_NETDMA && size &&sock_flag (sk,SOCK_ZEROCOPY )) { + zc = sk->sk_route_caps &NETIF_F_SG ; +if (!zc) { + err = -EFAULT ; + goto out_err; + } + } +if (unlikely (flags &MSG_FASTOPEN ||inet_sk (sk)->defer_connect) && !tp->repair) { err =tcp_sendmsg_fastopen (sk, msg, &copied_syn, size, uarg);
Signature · 1 line
2.24.1
RFC PATCH 19/21 core: add page recycling logic for netgpu pages
In reply to Jonathan Lemonnetgpu pages will always have a refcount of at least one (held by the netgpu module). This logic and the codepath obviously needs work, but suffices for a proof-of-concept.
--- net/core/skbuff.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
net/core/skbuff.c +25 -2
@@ -69,6 +69,7 @@#include <net/xfrm.h> #include <net/mpls.h> #include <net/mptcp.h> +#include <net/netgpu.h> #include <linux/uaccess.h> #include <trace/events/skb.h>
@@ -590,6 +591,24 @@ static void skb_free_head(struct sk_buff *skb)kfree (head); } +static void skb_netgpu_unref (struct skb_shared_info *shinfo) +{ +struct page *page; +int count; +int i; + +/* pages attached for skbs for TX shouldn't come here, since +* the skb is not marked as "zc_netgpu". (only RX skbs have this). +* dummy page does come here, but always has elevated refc. +*/ +for (i =0 ; i < shinfo->nr_frags; i++) { + page =skb_frag_page (&shinfo->frags[i]); + count =page_ref_dec_return (page); +if (count <=2 ) +__netgpu_put_page (g_ctx, page, false); + } +} +static void skb_release_data (struct sk_buff *skb) {struct skb_shared_info *shinfo =skb_shinfo (skb);
@@ -600,8 +619,12 @@ static void skb_release_data(struct sk_buff *skb) &shinfo->dataref))return ; -for (i =0 ; i < shinfo->nr_frags; i++) -__skb_frag_unref (&shinfo->frags[i]); +if (skb->zc_netgpu && shinfo->nr_frags) { +skb_netgpu_unref (shinfo); + }else { +for (i =0 ; i < shinfo->nr_frags; i++) +__skb_frag_unref (&shinfo->frags[i]); + }if (shinfo->frag_list)kfree_skb_list (shinfo->frag_list);
Signature · 1 line
2.24.1
kernel test robot
build botRe: [RFC PATCH 19/21] core: add page recycling logic for netgpu pages
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]
Hi Jonathan,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on net/master]
[also build test ERROR on linus/master v5.8-rc1 next-20200618] [cannot apply to char-misc/char-misc-testing net-next/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/netgpu-networking-between-NIC-and-GPU-CPU/20200619-001526 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git ef7232da6bcd4294cbb2d424bc35885721570f01 config: x86_64-kexec (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
ld: lib/iov_iter.o: in function `__netgpu_get_pages':
include/net/netgpu.h:64: undefined reference to `fn_netgpu_get_pages'
ld: include/net/netgpu.h:64: undefined reference to `fn_netgpu_get_pages'
ld: net/core/skbuff.o: in function `__netgpu_put_page':
>> include/net/netgpu.h:57: undefined reference to `g_ctx' >> ld: include/net/netgpu.h:57: undefined reference to `fn_netgpu_put_page'
vim +57 include/net/netgpu.h
60654acea6f2baf Jonathan Lemon 2020-06-18 53
60654acea6f2baf Jonathan Lemon 2020-06-18 54 static inline void
60654acea6f2baf Jonathan Lemon 2020-06-18 55 __netgpu_put_page(struct netgpu_ctx *ctx, struct page *page, bool napi)
60654acea6f2baf Jonathan Lemon 2020-06-18 56 {60654acea6f2baf Jonathan Lemon 2020-06-18 @57 return fn_netgpu_put_page(ctx, page, napi);60654acea6f2baf Jonathan Lemon 2020-06-18 58 }60654acea6f2baf Jonathan Lemon 2020-06-18 59
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
RFC PATCH 20/21 core/skbuff: use skb_zdata for testing whether skb is zerocopy
In reply to Jonathan Lemonskb_zcopy() flag indicates whether the skb has a zerocopy ubuf. netgpu does not use ubufs, so skb_zdata() indicates whether the skb is carrying zero copy data, and should be left alone, while skb_zcopy() indicates whhether there is an attached ubuf.
Also, when a write() on a zero-copy socket returns EWOULDBLOCK, this is not synchronized with select(), which will only look at the send buffer, and return writability if there is tcp space.
This appears to be caused by some ubuf logic, leading to iperf spending 70% of its time in select() for ZC transmits. With this change, the time spent drops to 20%.
--- include/linux/skbuff.h | 24 +++++++++++++++++++++++- net/core/skbuff.c | 16 ++++++++++++---- 2 files changed, 35 insertions(+), 5 deletions(-)
include/linux/skbuff.h +23 -1
@@ -443,8 +443,12 @@ enum {/* generate software time stamp when entering packet scheduling */ SKBTX_SCHED_TSTAMP =1 <<6 , + +/* fragments are accessed only via DMA */ + SKBTX_DEV_NETDMA = 1 << 7, }; +#define SKBTX_ZERODATA_FRAG (SKBTX_DEV_ZEROCOPY | SKBTX_DEV_NETDMA)#define SKBTX_ZEROCOPY_FRAG (SKBTX_DEV_ZEROCOPY | SKBTX_SHARED_FRAG)#define SKBTX_ANY_SW_TSTAMP (SKBTX_SW_TSTAMP | \ SKBTX_SCHED_TSTAMP)
@@ -1416,6 +1420,24 @@ static inline struct skb_shared_hwtstamps *skb_hwtstamps(struct sk_buff *skb)return &skb_shinfo (skb)->hwtstamps; } +static inline bool skb_netdma (struct sk_buff *skb) +{ +return skb &&skb_shinfo (skb)->tx_flags &SKBTX_DEV_NETDMA ; +} + +static inline bool skb_zdata (struct sk_buff *skb) +{ +return skb &&skb_shinfo (skb)->tx_flags &SKBTX_ZERODATA_FRAG ; +} + +static inline void skb_netdma_set (struct sk_buff *skb,bool netdma) +{ +if (skb && netdma) { +skb_shinfo (skb)->tx_flags |=SKBTX_DEV_NETDMA ; +skb_shinfo (skb)->destructor_arg =NULL ; + } +} +static inline struct ubuf_info *skb_zcopy (struct sk_buff *skb) {bool is_zcopy = skb &&skb_shinfo (skb)->tx_flags &SKBTX_DEV_ZEROCOPY ;
@@ -3260,7 +3282,7 @@ static inline int skb_add_data(struct sk_buff *skb,static inline bool skb_can_coalesce (struct sk_buff *skb,int i,const struct page *page,int off) { -if (skb_zcopy (skb)) +if (skb_zdata (skb))return false;if (i) {const skb_frag_t *frag = &skb_shinfo (skb)->frags[i -1 ];
net/core/skbuff.c +12 -4
@@ -1323,6 +1323,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, }skb_zcopy_set (skb, uarg,NULL ); +skb_netdma_set (skb, sk->sk_user_data); +return skb->len - orig_len; }EXPORT_SYMBOL_GPL (skb_zerocopy_iter_stream);
@@ -1330,8 +1332,8 @@ EXPORT_SYMBOL_GPL(skb_zerocopy_iter_stream);static int skb_zerocopy_clone (struct sk_buff *nskb,struct sk_buff *orig,gfp_t gfp_mask) { -if (skb_zcopy (orig)) { -if (skb_zcopy (nskb)) { +if (skb_zdata (orig)) { +if (skb_zdata (nskb)) {/* !gfp_mask callers are verified to !skb_zcopy(nskb) */ if (!gfp_mask) {WARN_ON_ONCE (1 );
@@ -1343,6 +1345,7 @@ static int skb_zerocopy_clone(struct sk_buff *nskb, struct sk_buff *orig,return -EIO ; }skb_zcopy_set (nskb,skb_uarg (orig),NULL ); +skb_netdma_set (nskb,skb_netdma (orig)); }return 0 ; }
@@ -1372,6 +1375,9 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)if (skb_shared (skb) ||skb_unclone (skb, gfp_mask))return -EINVAL ; +if (!skb_has_shared_frag (skb)) +return -EINVAL ; +if (!num_frags) goto release;
@@ -2078,6 +2084,8 @@ void *__pskb_pull_tail(struct sk_buff *skb, int delta)*/ int i, k, eat = (skb->tail + delta) - skb->end; +BUG_ON (skb_netdma (skb)); +if (eat >0 ||skb_cloned (skb)) {if (pskb_expand_head (skb,0 , eat >0 ? eat +128 :0 ,GFP_ATOMIC ))
@@ -3328,7 +3336,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)if (skb_headlen (skb))return 0 ; -if (skb_zcopy (tgt) ||skb_zcopy (skb)) +if (skb_zdata (tgt) ||skb_zdata (skb))return 0 ; todo = shiftlen;
@@ -5171,7 +5179,7 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, from_shinfo =skb_shinfo (from);if (to_shinfo->frag_list || from_shinfo->frag_list)return false; -if (skb_zcopy (to) ||skb_zcopy (from)) +if (skb_zdata (to) ||skb_zdata (from))return false;if (skb_headlen (from) !=0 ) {
Signature · 1 line
2.24.1
RFC PATCH 21/21 mlx5: add XDP_SETUP_NETGPU hook
In reply to Jonathan LemonAdd the hook which enables and disables the zero copy queues.
--- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
@@ -4598,6 +4598,9 @@ static int mlx5e_xdp(struct net_device *dev, struct netdev_bpf *xdp)case XDP_SETUP_XSK_UMEM :return mlx5e_xsk_setup_umem (dev, xdp->xsk.umem, xdp->xsk.queue_id); +case XDP_SETUP_NETGPU : +return mlx5e_netgpu_setup_ctx (dev, xdp->netgpu.ctx, + xdp->netgpu.queue_id);default :return -EINVAL ; }
Signature · 1 line
2.24.1