Skip to contentSkip to conversation

net: introduce Qualcomm IPA driver

PATCH In review Alex Elder 10 participants 106 messages May 31, 2019 – Jul 3, 2019

Subsystems (8)
lore
Share & tools
lore

Alex Elder

author
#0 lore raw

v2 cover net: introduce Qualcomm IPA driver

This series presents the driver for the Qualcomm IP Accelerator (IPA).

This is version 2 of the series. This version has addressed almost all of the feedback received in the first version:
https://lore.kernel.org/lkml/[email protected]/
More detail is included in the individual patches, but here is a high-level summary of what's changed since then:
- Two spinlocks have been removed.
- The code for enabling and disabling endpoint interrupts has been simplified considerably, and the spinlock is no longer required
- A spinlock used when updating ring buffer pointers is no longer needed. Integers indexing the ring are used instead (and they don't even have to be atomic).
- One spinlock remains to protect list updates, but it is always acquired using spin_lock_bh() (no more irqsave).
- Information about the queueing and completion of messages is now supplied to the network stack in batches rather than one at a time.
- I/O completion handling has been simplified, with the IRQ handler now consisting mainly of disabling the interrupt and calling napi_schedule().
- Some comments have been updated and improved througout.

What follows is the introduction supplied with v1 of the series.

-----

The IPA is a component present in some Qualcomm SoCs that allows network functions such as aggregation, filtering, routing, and NAT to be performed without active involvement of the main application processor (AP).

Initially, these advanced features are disabled; the IPA driver simply provides a network interface that makes the modem's LTE network available to the AP. In addition, only support for the IPA found in the Qualcomm SDM845 SoC is provided.

This code is derived from a driver developed internally by Qualcomm. A version of the original source can be seen here:
https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree
in the "drivers/platform/msm/ipa" directory. Many were involved in developing this, but the following individuals deserve explicit acknowledgement for their substantial contributions:

Abhishek Choubey
Ady Abraham
Chaitanya Pratapa
David Arinzon
Ghanim Fodi
Gidon Studinski
Ravi Gummadidala
Shihuan Liu
Skylar Chang

A version of this code was posted in November 2018 as an RFC.
https://lore.kernel.org/lkml/[email protected]/
All feedback received was addressed. The code has undergone considerable further rework since that time, and most of the "future work" described then has now been completed.

This code is available in buildable form here, based on kernel v5.2-rc1:
remote: ssh://[email protected]/people/alex.elder/linux.git
branch: ipa-v2_kernel-v5.2-rc2
75adf2ac1266 arm64: defconfig: enable build of IPA code

The branch depends on a commit now found in in net-next. It has been cherry-picked, and (in this branch) has this commit ID:
13c627b5a078 net: qualcomm: rmnet: Move common struct definitions to include
by

-Alex

Alex Elder (17):
bitfield.h: add FIELD_MAX() and field_max()
dt-bindings: soc: qcom: add IPA bindings
soc: qcom: ipa: main code
soc: qcom: ipa: configuration data
soc: qcom: ipa: clocking, interrupts, and memory
soc: qcom: ipa: GSI headers
soc: qcom: ipa: the generic software interface
soc: qcom: ipa: GSI transactions
soc: qcom: ipa: IPA interface to GSI
soc: qcom: ipa: IPA endpoints
soc: qcom: ipa: immediate commands
soc: qcom: ipa: IPA network device and microcontroller
soc: qcom: ipa: AP/modem communications
soc: qcom: ipa: support build of IPA code
MAINTAINERS: add entry for the Qualcomm IPA driver
arm64: dts: sdm845: add IPA information
arm64: defconfig: enable build of IPA code

 .../devicetree/bindings/net/qcom,ipa.yaml     |  180 ++
 MAINTAINERS                                   |    6 +
 arch/arm64/boot/dts/qcom/sdm845.dtsi          |   51 +
 arch/arm64/configs/defconfig                  |    1 +
 drivers/net/Kconfig                           |    2 +
 drivers/net/Makefile                          |    1 +
 drivers/net/ipa/Kconfig                       |   16 +
 drivers/net/ipa/Makefile                      |    7 +
 drivers/net/ipa/gsi.c                         | 1635 +++++++++++++++++
 drivers/net/ipa/gsi.h                         |  246 +++
 drivers/net/ipa/gsi_private.h                 |  148 ++
 drivers/net/ipa/gsi_reg.h                     |  376 ++++
 drivers/net/ipa/gsi_trans.c                   |  624 +++++++
 drivers/net/ipa/gsi_trans.h                   |  116 ++
 drivers/net/ipa/ipa.h                         |  131 ++
 drivers/net/ipa/ipa_clock.c                   |  297 +++
 drivers/net/ipa/ipa_clock.h                   |   52 +
 drivers/net/ipa/ipa_cmd.c                     |  377 ++++
 drivers/net/ipa/ipa_cmd.h                     |  116 ++
 drivers/net/ipa/ipa_data-sdm845.c             |  245 +++
 drivers/net/ipa/ipa_data.h                    |  267 +++
 drivers/net/ipa/ipa_endpoint.c                | 1283 +++++++++++++
 drivers/net/ipa/ipa_endpoint.h                |   97 +
 drivers/net/ipa/ipa_gsi.c                     |   48 +
 drivers/net/ipa/ipa_gsi.h                     |   49 +
 drivers/net/ipa/ipa_interrupt.c               |  279 +++
 drivers/net/ipa/ipa_interrupt.h               |   53 +
 drivers/net/ipa/ipa_main.c                    |  921 ++++++++++
 drivers/net/ipa/ipa_mem.c                     |  234 +++
 drivers/net/ipa/ipa_mem.h                     |   83 +
 drivers/net/ipa/ipa_netdev.c                  |  251 +++
 drivers/net/ipa/ipa_netdev.h                  |   24 +
 drivers/net/ipa/ipa_qmi.c                     |  402 ++++
 drivers/net/ipa/ipa_qmi.h                     |   35 +
 drivers/net/ipa/ipa_qmi_msg.c                 |  583 ++++++
 drivers/net/ipa/ipa_qmi_msg.h                 |  238 +++
 drivers/net/ipa/ipa_reg.h                     |  279 +++
 drivers/net/ipa/ipa_smp2p.c                   |  304 +++
 drivers/net/ipa/ipa_smp2p.h                   |   47 +
 drivers/net/ipa/ipa_uc.c                      |  208 +++
 drivers/net/ipa/ipa_uc.h                      |   32 +
 include/linux/bitfield.h                      |   14 +
 42 files changed, 10358 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml
 create mode 100644 drivers/net/ipa/Kconfig
 create mode 100644 drivers/net/ipa/Makefile
 create mode 100644 drivers/net/ipa/gsi.c
 create mode 100644 drivers/net/ipa/gsi.h
 create mode 100644 drivers/net/ipa/gsi_private.h
 create mode 100644 drivers/net/ipa/gsi_reg.h
 create mode 100644 drivers/net/ipa/gsi_trans.c
 create mode 100644 drivers/net/ipa/gsi_trans.h
 create mode 100644 drivers/net/ipa/ipa.h
 create mode 100644 drivers/net/ipa/ipa_clock.c
 create mode 100644 drivers/net/ipa/ipa_clock.h
 create mode 100644 drivers/net/ipa/ipa_cmd.c
 create mode 100644 drivers/net/ipa/ipa_cmd.h
 create mode 100644 drivers/net/ipa/ipa_data-sdm845.c
 create mode 100644 drivers/net/ipa/ipa_data.h
 create mode 100644 drivers/net/ipa/ipa_endpoint.c
 create mode 100644 drivers/net/ipa/ipa_endpoint.h
 create mode 100644 drivers/net/ipa/ipa_gsi.c
 create mode 100644 drivers/net/ipa/ipa_gsi.h
 create mode 100644 drivers/net/ipa/ipa_interrupt.c
 create mode 100644 drivers/net/ipa/ipa_interrupt.h
 create mode 100644 drivers/net/ipa/ipa_main.c
 create mode 100644 drivers/net/ipa/ipa_mem.c
 create mode 100644 drivers/net/ipa/ipa_mem.h
 create mode 100644 drivers/net/ipa/ipa_netdev.c
 create mode 100644 drivers/net/ipa/ipa_netdev.h
 create mode 100644 drivers/net/ipa/ipa_qmi.c
 create mode 100644 drivers/net/ipa/ipa_qmi.h
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.c
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.h
 create mode 100644 drivers/net/ipa/ipa_reg.h
 create mode 100644 drivers/net/ipa/ipa_smp2p.c
 create mode 100644 drivers/net/ipa/ipa_smp2p.h
 create mode 100644 drivers/net/ipa/ipa_uc.c
 create mode 100644 drivers/net/ipa/ipa_uc.h
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v2 1/17 bitfield.h: add FIELD_MAX() and field_max()

Define FIELD_MAX(), which supplies the maximum value that can be represented by a field value. Define field_max() as well, to go along with the lower-case forms of the field mask functions.

---
 include/linux/bitfield.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
include/linux/bitfield.h +14 -0
index 3f1ef4450a7c..cf4f06774520 100644
@@ -63,6 +63,19 @@
 					      (1ULL << __bf_shf(_mask))); \
 	})
 
+/**
+ * FIELD_MAX() - produce the maximum value representable by a field
+ * @_mask: shifted mask defining the field's length and position
+ *
+ * FIELD_MAX() returns the maximum value that can be held in the field
+ * specified by @_mask.
+ */
+#define FIELD_MAX(_mask)						\
+	({								\
+		__BF_FIELD_CHECK(_mask, 0ULL, 0ULL, "FIELD_MAX: ");	\
+		(typeof(_mask))((_mask) >> __bf_shf(_mask));		\
+	})
+
 /**
  * FIELD_FIT() - check if value fits in the field
  * @_mask: shifted mask defining the field's length and position
@@ -118,6 +131,7 @@ static __always_inline u64 field_mask(u64 field)
 {
 	return field / field_multiplier(field);
 }
+#define field_max(field)	((typeof(field))field_mask(field))
 #define ____MAKE_OP(type,base,to,from)					\
 static __always_inline __##type type##_encode_bits(base v, base field)	\
 {									\
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH v2 2/17 dt-bindings: soc: qcom: add IPA bindings

In reply to Alex Elder

Add the binding definitions for the "qcom,ipa" device tree node.

---
 .../devicetree/bindings/net/qcom,ipa.yaml     | 180 ++++++++++++++++++
 1 file changed, 180 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml
Documentation/devicetree/bindings/net/qcom,ipa.yaml new file +180 -0
new file mode 100644
index 000000000000..0037fc278a61
@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: GPL-2.0
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/net/qcom,ipa.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm IP Accelerator (IPA)
+
+maintainers:
+  - Alex Elder <[email protected]>
+
+description:
+  This binding describes the Qualcomm IPA.  The IPA is capable of offloading
+  certain network processing tasks (e.g. filtering, routing, and NAT) from
+  the main processor.
+
+  The IPA sits between multiple independent "execution environments,"
+  including the Application Processor (AP) and the modem.  The IPA presents
+  a Generic Software Interface (GSI) to each execution environment.
+  The GSI is an integral part of the IPA, but it is logically isolated
+  and has a distinct interrupt and a separately-defined address space.
+
+  See also soc/qcom/qcom,smp2p.txt and interconnect/interconnect.txt.
+
+  - |
+    --------             ---------
+    |      |             |       |
+    |  AP  +<---.   .----+ Modem |
+    |      +--. |   | .->+       |
+    |      |  | |   | |  |       |
+    --------  | |   | |  ---------
+              v |   v |
+            --+-+---+-+--
+            |    GSI    |
+            |-----------|
+            |           |
+            |    IPA    |
+            |           |
+            -------------
+
+properties:
+  compatible:
+      const: "qcom,sdm845-ipa"
+
+  reg:
+    items:
+      - description: IPA registers
+      - description: IPA shared memory
+      - description: GSI registers
+
+  reg-names:
+    items:
+      - const: ipa-reg
+      - const: ipa-shared
+      - const: gsi
+
+  clocks:
+    maxItems: 1
+
+  clock-names:
+      const: core
+
+  interrupts:
+    items:
+      - description: IPA interrupt (hardware IRQ)
+      - description: GSI interrupt (hardware IRQ)
+      - description: Modem clock query interrupt (smp2p interrupt)
+      - description: Modem setup ready interrupt (smp2p interrupt)
+
+  interrupt-names:
+    items:
+      - const: ipa
+      - const: gsi
+      - const: ipa-clock-query
+      - const: ipa-setup-ready
+
+  interconnects:
+    items:
+      - description: Interconnect path between IPA and main memory
+      - description: Interconnect path between IPA and internal memory
+      - description: Interconnect path between IPA and the AP subsystem
+
+  interconnect-names:
+    items:
+      - const: memory
+      - const: imem
+      - const: config
+
+  qcom,smem-states:
+    description: State bits used in by the AP to signal the modem.
+    items:
+    - description: Whether the "ipa-clock-enabled" state bit is valid
+    - description: Whether the IPA clock is enabled (if valid)
+
+  qcom,smem-state-names:
+    description: The names of the state bits used for SMP2P output
+    items:
+      - const: ipa-clock-enabled-valid
+      - const: ipa-clock-enabled
+
+  modem-init:
+    type: boolean
+    description:
+      If present, it indicates that the modem is responsible for
+      performing early IPA initialization, including loading and
+      validating firwmare used by the GSI.
+
+  memory-region:
+    maxItems: 1
+    description:
+      If present, a phandle for a reserved memory area that holds
+      the firmware passed to Trust Zone for authentication.  Required
+      when Trust Zone (not the modem) performs early initialization.
+
+required:
+  - compatible
+  - reg
+  - clocks
+  - interrupts
+  - interconnects
+  - qcom,smem-states
+
+oneOf:
+  - required:
+    - modem-init
+  - required:
+    - memory-region
+
+examples:
+  - |
+        smp2p-mpss {
+                compatible = "qcom,smp2p";
+                ipa_smp2p_out: ipa-ap-to-modem {
+                        qcom,entry-name = "ipa";
+                        #qcom,smem-state-cells = <1>;
+                };
+
+                ipa_smp2p_in: ipa-modem-to-ap {
+                        qcom,entry-name = "ipa";
+                        interrupt-controller;
+                        #interrupt-cells = <2>;
+                };
+        };
+        ipa@1e40000 {
+                compatible = "qcom,sdm845-ipa";
+
+                modem-init;
+
+                reg = <0 0x1e40000 0 0x7000>,
+                        <0 0x1e47000 0 0x2000>,
+                        <0 0x1e04000 0 0x2c000>;
+                reg-names = "ipa-reg",
+                                "ipa-shared";
+                                "gsi";
+
+                interrupts-extended = <&intc 0 311 IRQ_TYPE_EDGE_RISING>,
+                                        <&intc 0 432 IRQ_TYPE_LEVEL_HIGH>,
+                                        <&ipa_smp2p_in 0 IRQ_TYPE_EDGE_RISING>,
+                                        <&ipa_smp2p_in 1 IRQ_TYPE_EDGE_RISING>;
+                interrupt-names = "ipa",
+                                        "gsi",
+                                        "ipa-clock-query",
+                                        "ipa-setup-ready";
+
+                clocks = <&rpmhcc RPMH_IPA_CLK>;
+                clock-names = "core";
+
+                interconnects =
+                        <&rsc_hlos MASTER_IPA &rsc_hlos SLAVE_EBI1>,
+                        <&rsc_hlos MASTER_IPA &rsc_hlos SLAVE_IMEM>,
+                        <&rsc_hlos MASTER_APPSS_PROC &rsc_hlos SLAVE_IPA_CFG>;
+                interconnect-names = "memory",
+                                        "imem",
+                                        "config";
+
+                qcom,smem-states = <&ipa_smp2p_out 0>,
+                                        <&ipa_smp2p_out 1>;
+                qcom,smem-state-names = "ipa-clock-enabled-valid",
+                                        "ipa-clock-enabled";
+        };
Signature · 1 line
2.20.1
author
#6 lore raw

PATCH v2 3/17 soc: qcom: ipa: main code

In reply to Alex Elder

This patch includes three source files that represent some basic "main program" code for the IPA driver. They are:
- "ipa.h" defines the top-level IPA structure which represents an IPA device throughout the code.
- "ipa_main.c" contains the platform driver probe function, along with some general code used during initialization.
- "ipa_reg.h" defines the offsets of the 32-bit registers used for the IPA device, along with masks that define the position and width of fields less than 32 bits located within these registers.

Each file includes some documentation that provides a little more overview of how the code is organized and used.

---
 drivers/net/ipa/ipa.h      | 131 ++++++
 drivers/net/ipa/ipa_main.c | 921 +++++++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_reg.h  | 279 +++++++++++
 3 files changed, 1331 insertions(+)
 create mode 100644 drivers/net/ipa/ipa.h
 create mode 100644 drivers/net/ipa/ipa_main.c
 create mode 100644 drivers/net/ipa/ipa_reg.h
drivers/net/ipa/ipa.h new file +131 -0
new file mode 100644
index 000000000000..c580254d1e0e
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+#ifndef _IPA_H_
+#define _IPA_H_
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/notifier.h>
+#include <linux/pm_wakeup.h>
+
+#include "gsi.h"
+#include "ipa_qmi.h"
+#include "ipa_endpoint.h"
+#include "ipa_interrupt.h"
+
+struct clk;
+struct icc_path;
+struct net_device;
+struct platform_device;
+
+struct ipa_clock;
+struct ipa_smp2p;
+struct ipa_interrupt;
+
+/**
+ * struct ipa - IPA information
+ * @gsi:		Embedded GSI structure
+ * @pdev:		Platform device
+ * @smp2p:		SMP2P information
+ * @clock:		IPA clocking information
+ * @suspend_ref:	Whether clock reference preventing suspend taken
+ * @route_virt:		Virtual address of routing table
+ * @route_addr:		DMA address for routing table
+ * @filter_virt:	Virtual address of filter table
+ * @filter_addr:	DMA address for filter table
+ * @interrupt:		IPA Interrupt information
+ * @uc_loaded:		Non-zero when microcontroller has reported it's ready
+ * @ipa_phys:		Physical address of IPA memory space
+ * @ipa_virt:		Virtual address for IPA memory space
+ * @reg_virt:		Virtual address used for IPA register access
+ * @shared_phys:	Physical address of memory space shared with modem
+ * @shared_virt:	Virtual address of memory space shared with modem
+ * @shared_offset:	Additional offset used for shared memory
+ * @wakeup:		Wakeup source information
+ * @filter_support:	Bit mask indicating endpoints that support filtering
+ * @initialized:	Bit mask indicating endpoints initialized
+ * @set_up:		Bit mask indicating endpoints set up
+ * @enabled:		Bit mask indicating endpoints enabled
+ * @suspended:		Bit mask indicating endpoints suspended
+ * @endpoint:		Array of endpoint information
+ * @endpoint_map:	Mapping of GSI channel to IPA endpoint information
+ * @command_endpoint:	Endpoint used for command TX
+ * @default_endpoint:	Endpoint used for default route RX
+ * @modem_netdev:	Network device structure used for modem
+ * @setup_complete:	Flag indicating whether setup stage has completed
+ * @qmi:		QMI information
+ */
+struct ipa {
+	struct gsi gsi;
+	struct platform_device *pdev;
+	struct ipa_smp2p *smp2p;
+	struct ipa_clock *clock;
+	atomic_t suspend_ref;
+
+	void *route_virt;
+	dma_addr_t route_addr;
+	void *filter_virt;
+	dma_addr_t filter_addr;
+
+	struct ipa_interrupt *interrupt;
+	u32 uc_loaded;
+
+	phys_addr_t reg_phys;
+	void __iomem *reg_virt;
+	phys_addr_t shared_phys;
+	void *shared_virt;
+	u32 shared_offset;
+
+	struct wakeup_source wakeup;
+
+	/* Bit masks indicating endpoint state */
+	u32 filter_support;
+	u32 initialized;
+	u32 set_up;
+	u32 enabled;
+	u32 suspended;
+
+	struct ipa_endpoint endpoint[IPA_ENDPOINT_MAX];
+	struct ipa_endpoint *endpoint_map[GSI_CHANNEL_MAX];
+	struct ipa_endpoint *command_endpoint;	/* TX */
+	struct ipa_endpoint *default_endpoint;	/* Default route RX */
+
+	struct net_device *modem_netdev;
+	u32 setup_complete;
+
+	struct ipa_qmi qmi;
+};
+
+/**
+ * ipa_setup() - Perform IPA setup
+ * @ipa:		IPA pointer
+ *
+ * IPA initialization is broken into stages:  init; config; setup; and
+ * sometimes enable.  (These have inverses exit, deconfig, teardown, and
+ * disable.)  Activities performed at the init stage can be done without
+ * requiring any access to hardware.  For IPA, activities performed at the
+ * config stage require the IPA clock to be running, because they involve
+ * access to IPA registers.  The setup stage is performed only after the
+ * GSI hardware is ready (more on this below).  And finally IPA endpoints
+ * can be enabled once they're successfully set up.
+ *
+ * This function, @ipa_setup(), starts the setup stage.
+ *
+ * In order for the GSI hardware to be functional it needs firmware to be
+ * loaded (in addition to some other low-level initialization).  This early
+ * GSI initialization can be done either by Trust Zone or by the modem.  If
+ * it's done by Trust Zone, the AP loads the GSI firmware and supplies it to
+ * Trust Zone to verify and install.  The AP knows when this completes, and
+ * whether it was successful.  In this case the AP proceeds to setup once it
+ * knows GSI is ready.
+ *
+ * If the modem performs early GSI initialization, the AP needs to know when
+ * this has occurred.  An SMP2P interrupt is used for this purpose, and
+ * receipt of that interrupt triggers the call to ipa_setup().
+ */
+int ipa_setup(struct ipa *ipa);
+
+#endif /* _IPA_H_ */
drivers/net/ipa/ipa_main.c new file +921 -0

921 lines not on this page · raw message

drivers/net/ipa/ipa_reg.h new file +279 -0

279 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#9 lore raw

PATCH v2 4/17 soc: qcom: ipa: configuration data

In reply to Alex Elder

This patch defines configuration data that is used to specify some of the details of IPA hardware supported by the driver. It is built as Device Tree match data, discovered at boot time. Initially the driver only supports the Qualcomm SDM845 SoC.

---
 drivers/net/ipa/ipa_data-sdm845.c | 245 +++++++++++++++++++++++++++
 drivers/net/ipa/ipa_data.h        | 267 ++++++++++++++++++++++++++++++
 2 files changed, 512 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_data-sdm845.c
 create mode 100644 drivers/net/ipa/ipa_data.h
drivers/net/ipa/ipa_data-sdm845.c new file +245 -0
new file mode 100644
index 000000000000..62c0f25f5161
@@ -0,0 +1,245 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019 Linaro Ltd.
+ */
+
+#include <linux/log2.h>
+
+#include "gsi.h"
+#include "ipa_data.h"
+#include "ipa_endpoint.h"
+
+/* Differentiate Boolean from numerical options */
+#define NO	0
+#define YES	1
+
+/* Endpoint configuration for the SDM845 SoC. */
+static const struct gsi_ipa_endpoint_data gsi_ipa_endpoint_data[] = {
+	{
+		.ee_id		= GSI_EE_AP,
+		.channel_id	= 4,
+		.endpoint_id	= IPA_ENDPOINT_AP_COMMAND_TX,
+		.toward_ipa	= YES,
+		.channel = {
+			.tlv_count	= 20,
+			.wrr_priority	= YES,
+			.tre_count	= 256,
+			.event_count	= 512,
+		},
+		.endpoint = {
+			.seq_type	= IPA_SEQ_DMA_ONLY,
+			.config = {
+				.dma_mode	= YES,
+				.dma_endpoint	= IPA_ENDPOINT_AP_LAN_RX,
+			},
+		},
+	},
+	{
+		.ee_id		= GSI_EE_AP,
+		.channel_id	= 5,
+		.endpoint_id	= IPA_ENDPOINT_AP_LAN_RX,
+		.toward_ipa	= NO,
+		.channel = {
+			.tlv_count	= 8,
+			.tre_count	= 256,
+			.event_count	= 256,
+		},
+		.endpoint = {
+			.seq_type	= IPA_SEQ_INVALID,
+			.config = {
+				.checksum	= YES,
+				.aggregation	= YES,
+				.status_enable	= YES,
+				.rx = {
+					.pad_align	= ilog2(sizeof(u32)),
+				},
+			},
+		},
+	},
+	{
+		.ee_id		= GSI_EE_AP,
+		.channel_id	= 3,
+		.endpoint_id	= IPA_ENDPOINT_AP_MODEM_TX,
+		.toward_ipa	= YES,
+		.channel = {
+			.tlv_count	= 16,
+			.tre_count	= 512,
+			.event_count	= 512,
+		},
+		.endpoint = {
+			.support_flt	= YES,
+			.seq_type	=
+				IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+			.config = {
+				.checksum	= YES,
+				.qmap		= YES,
+				.status_enable	= YES,
+				.tx = {
+					.delay	= YES,
+					.status_endpoint =
+						IPA_ENDPOINT_MODEM_AP_RX,
+				},
+			},
+		},
+	},
+	{
+		.ee_id		= GSI_EE_AP,
+		.channel_id	= 6,
+		.endpoint_id	= IPA_ENDPOINT_AP_MODEM_RX,
+		.toward_ipa	= NO,
+		.channel = {
+			.tlv_count	= 8,
+			.tre_count	= 256,
+			.event_count	= 256,
+		},
+		.endpoint = {
+			.seq_type	= IPA_SEQ_INVALID,
+			.config = {
+				.checksum	= YES,
+				.qmap		= YES,
+				.aggregation	= YES,
+				.rx = {
+					.aggr_close_eof	= YES,
+				},
+			},
+		},
+	},
+	{
+		.ee_id		= GSI_EE_MODEM,
+		.channel_id	= 1,
+		.endpoint_id	= IPA_ENDPOINT_MODEM_COMMAND_TX,
+		.toward_ipa	= YES,
+		.endpoint = {
+			.seq_type	= IPA_SEQ_PKT_PROCESS_NO_DEC_UCP,
+		},
+	},
+	{
+		.ee_id		= GSI_EE_MODEM,
+		.channel_id	= 0,
+		.endpoint_id	= IPA_ENDPOINT_MODEM_LAN_TX,
+		.toward_ipa	= YES,
+		.endpoint = {
+			.support_flt	= YES,
+		},
+	},
+	{
+		.ee_id		= GSI_EE_MODEM,
+		.channel_id	= 3,
+		.endpoint_id	= IPA_ENDPOINT_MODEM_LAN_RX,
+		.toward_ipa	= NO,
+	},
+	{
+		.ee_id		= GSI_EE_MODEM,
+		.channel_id	= 4,
+		.endpoint_id	= IPA_ENDPOINT_MODEM_AP_TX,
+		.toward_ipa	= YES,
+		.endpoint = {
+			.support_flt	= YES,
+		},
+	},
+	{
+		.ee_id		= GSI_EE_MODEM,
+		.channel_id	= 2,
+		.endpoint_id	= IPA_ENDPOINT_MODEM_AP_RX,
+		.toward_ipa	= NO,
+	},
+};
+
+static const struct ipa_resource_src ipa_resource_src[] = {
+	{
+		.type = IPA_RESOURCE_TYPE_SRC_PKT_CONTEXTS,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 1,
+			.max = 63,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 1,
+			.max = 63,
+		},
+	},
+	{
+		.type = IPA_RESOURCE_TYPE_SRC_DESCRIPTOR_LISTS,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 10,
+			.max = 10,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 10,
+			.max = 10,
+		},
+	},
+	{
+		.type = IPA_RESOURCE_TYPE_SRC_DESCRIPTOR_BUFF,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 12,
+			.max = 12,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 14,
+			.max = 14,
+		},
+	},
+	{
+		.type = IPA_RESOURCE_TYPE_SRC_HPS_DMARS,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 0,
+			.max = 63,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 0,
+			.max = 63,
+		},
+	},
+	{
+		.type = IPA_RESOURCE_TYPE_SRC_ACK_ENTRIES,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 14,
+			.max = 14,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 20,
+			.max = 20,
+		},
+	},
+};
+
+static const struct ipa_resource_dst ipa_resource_dst[] = {
+	{
+		.type = IPA_RESOURCE_TYPE_DST_DATA_SECTORS,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 4,
+			.max = 4,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 4,
+			.max = 4,
+		},
+	},
+	{
+		.type = IPA_RESOURCE_TYPE_DST_DPS_DMARS,
+		.limits[IPA_RESOURCE_GROUP_LWA_DL] = {
+			.min = 2,
+			.max = 63,
+		},
+		.limits[IPA_RESOURCE_GROUP_UL_DL] = {
+			.min = 1,
+			.max = 63,
+		},
+	},
+};
+
+/* Resource configuration for the SDM845 SoC. */
+static const struct ipa_resource_data ipa_resource_data = {
+	.resource_src		= ipa_resource_src,
+	.resource_src_count	= ARRAY_SIZE(ipa_resource_src),
+	.resource_dst		= ipa_resource_dst,
+	.resource_dst_count	= ARRAY_SIZE(ipa_resource_dst),
+};
+
+/* Configuration data for the SDM845 SoC. */
+const struct ipa_data ipa_data_sdm845 = {
+	.endpoint_data		= gsi_ipa_endpoint_data,
+	.endpoint_data_count	= ARRAY_SIZE(gsi_ipa_endpoint_data),
+	.resource_data		= &ipa_resource_data,
+};
drivers/net/ipa/ipa_data.h new file +267 -0

267 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#10 lore raw

PATCH v2 5/17 soc: qcom: ipa: clocking, interrupts, and memory

In reply to Alex Elder

This patch incorporates three source files (and their headers). They're grouped into one patch mainly for the purpose of making the number and size of patches in this series somewhat reasonable.

- "ipa_clock.c" and "ipa_clock.h" implement clocking for the IPA device. The IPA has a single core clock managed by the common clock framework. In addition, the IPA has three buses whose bandwidth is managed by the Linux interconnect framework. At this time the core clock and all three buses are either on or off; we don't yet do any more fine-grained management than that. The core clock and interconnects are enabled and disabled as a unit, using a unified clock-like abstraction, ipa_clock_get()/ipa_clock_put().

- "ipa_interrupt.c" and "ipa_interrupt.h" implement IPA interrupts. There are two hardare IRQs used by the IPA driver (the other is the GSI interrupt, described in a separate patch). Several types of interrupt are handled by the IPA IRQ handler; these are not part of data/fast path.

- The IPA has a region of local memory that is accessible by the AP (and modem). Within that region are areas with certain defined purposes. "ipa_mem.c" and "ipa_mem.h" define those regions, and implement their initialization.

---
 drivers/net/ipa/ipa_clock.c     | 297 ++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_clock.h     |  52 ++++++
 drivers/net/ipa/ipa_interrupt.c | 279 ++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_interrupt.h |  53 ++++++
 drivers/net/ipa/ipa_mem.c       | 234 +++++++++++++++++++++++++
 drivers/net/ipa/ipa_mem.h       |  83 +++++++++
 6 files changed, 998 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_clock.c
 create mode 100644 drivers/net/ipa/ipa_clock.h
 create mode 100644 drivers/net/ipa/ipa_interrupt.c
 create mode 100644 drivers/net/ipa/ipa_interrupt.h
 create mode 100644 drivers/net/ipa/ipa_mem.c
 create mode 100644 drivers/net/ipa/ipa_mem.h
drivers/net/ipa/ipa_clock.c new file +297 -0
new file mode 100644
index 000000000000..9ed12e8183ad
@@ -0,0 +1,297 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+
+#include <linux/atomic.h>
+#include <linux/mutex.h>
+#include <linux/clk.h>
+#include <linux/device.h>
+#include <linux/interconnect.h>
+
+#include "ipa.h"
+#include "ipa_clock.h"
+#include "ipa_netdev.h"
+
+/**
+ * DOC: IPA Clocking
+ *
+ * The "IPA Clock" manages both the IPA core clock and the interconnects
+ * (buses) the IPA depends on as a single logical entity.  A reference count
+ * is incremented by "get" operations and decremented by "put" operations.
+ * Transitions of that count from 0 to 1 result in the clock and interconnects
+ * being enabled, and transitions of the count from 1 to 0 cause them to be
+ * disabled.  We currently operate the core clock at a fixed clock rate, and
+ * all buses at a fixed average and peak bandwidth.  As more advanced IPA
+ * features are enabled, we can will better use of clock and bus scaling.
+ *
+ * An IPA clock reference must be held for any access to IPA hardware.
+ */
+
+#define	IPA_CORE_CLOCK_RATE		(75UL * 1000 * 1000)	/* Hz */
+
+/* Interconnect path bandwidths (each times 1000 bytes per second) */
+#define IPA_MEMORY_AVG			(80 * 1000)	/* 80 MBps */
+#define IPA_MEMORY_PEAK			(600 * 1000)
+
+#define IPA_IMEM_AVG			(80 * 1000)
+#define IPA_IMEM_PEAK			(350 * 1000)
+
+#define IPA_CONFIG_AVG			(40 * 1000)
+#define IPA_CONFIG_PEAK			(40 * 1000)
+
+/**
+ * struct ipa_clock - IPA clocking information
+ * @core:		IPA core clock
+ * @memory_path:	Memory interconnect
+ * @imem_path:		Internal memory interconnect
+ * @config_path:	Configuration space interconnect
+ * @mutex;		Protects clock enable/disable
+ * @count:		Clocking reference count
+ */
+struct ipa_clock {
+	struct ipa *ipa;
+	atomic_t count;
+	struct mutex mutex; /* protects clock enable/disable */
+	struct clk *core;
+	struct icc_path *memory_path;
+	struct icc_path *imem_path;
+	struct icc_path *config_path;
+};
+
+/* Initialize interconnects required for IPA operation */
+static int ipa_interconnect_init(struct ipa_clock *clock, struct device *dev)
+{
+	struct icc_path *path;
+
+	path = of_icc_get(dev, "memory");
+	if (IS_ERR(path))
+		goto err_return;
+	clock->memory_path = path;
+
+	path = of_icc_get(dev, "imem");
+	if (IS_ERR(path))
+		goto err_memory_path_put;
+	clock->imem_path = path;
+
+	path = of_icc_get(dev, "config");
+	if (IS_ERR(path))
+		goto err_imem_path_put;
+	clock->config_path = path;
+
+	return 0;
+
+err_imem_path_put:
+	icc_put(clock->imem_path);
+err_memory_path_put:
+	icc_put(clock->memory_path);
+err_return:
+
+	return PTR_ERR(path);
+}
+
+/* Inverse of ipa_interconnect_init() */
+static void ipa_interconnect_exit(struct ipa_clock *clock)
+{
+	icc_put(clock->config_path);
+	icc_put(clock->imem_path);
+	icc_put(clock->memory_path);
+}
+
+/* Currently we only use one bandwidth level, so just "enable" interconnects */
+static int ipa_interconnect_enable(struct ipa_clock *clock)
+{
+	int ret;
+
+	ret = icc_set_bw(clock->memory_path, IPA_MEMORY_AVG, IPA_MEMORY_PEAK);
+	if (ret)
+		return ret;
+
+	ret = icc_set_bw(clock->imem_path, IPA_IMEM_AVG, IPA_IMEM_PEAK);
+	if (ret)
+		goto err_disable_memory_path;
+
+	ret = icc_set_bw(clock->config_path, IPA_CONFIG_AVG, IPA_CONFIG_PEAK);
+	if (ret)
+		goto err_disable_imem_path;
+
+	return 0;
+
+err_disable_imem_path:
+	(void)icc_set_bw(clock->imem_path, 0, 0);
+err_disable_memory_path:
+	(void)icc_set_bw(clock->memory_path, 0, 0);
+
+	return ret;
+}
+
+/* To disable an interconnect, we just its bandwidth to 0 */
+static int ipa_interconnect_disable(struct ipa_clock *clock)
+{
+	int ret;
+
+	ret = icc_set_bw(clock->memory_path, 0, 0);
+	if (ret)
+		return ret;
+
+	ret = icc_set_bw(clock->imem_path, 0, 0);
+	if (ret)
+		goto err_reenable_memory_path;
+
+	ret = icc_set_bw(clock->config_path, 0, 0);
+	if (ret)
+		goto err_reenable_imem_path;
+
+	return 0;
+
+err_reenable_imem_path:
+	(void)icc_set_bw(clock->imem_path, IPA_IMEM_AVG, IPA_IMEM_PEAK);
+err_reenable_memory_path:
+	(void)icc_set_bw(clock->memory_path, IPA_MEMORY_AVG, IPA_MEMORY_PEAK);
+
+	return ret;
+}
+
+/* Turn on IPA clocks, including interconnects */
+static int ipa_clock_enable(struct ipa_clock *clock)
+{
+	int ret;
+
+	ret = ipa_interconnect_enable(clock);
+	if (ret)
+		return ret;
+
+	ret = clk_prepare_enable(clock->core);
+	if (ret)
+		ipa_interconnect_disable(clock);
+
+	return ret;
+}
+
+/* Inverse of ipa_clock_enable() */
+static void ipa_clock_disable(struct ipa_clock *clock)
+{
+	clk_disable_unprepare(clock->core);
+	(void)ipa_interconnect_disable(clock);
+}
+
+/* Get an IPA clock reference, but only if the reference count is
+ * already non-zero.  Returns true if the additional reference was
+ * added successfully, or false otherwise.
+ */
+bool ipa_clock_get_additional(struct ipa_clock *clock)
+{
+	return !!atomic_inc_not_zero(&clock->count);
+}
+
+/* Get an IPA clock reference.  If the reference count is non-zero, it is
+ * incremented and return is immediate.  Otherwise it is checked again
+ * under protection of the mutex, and enable clocks and resume RX endpoints
+ * before returning.  For the first reference, the count is intentionally
+ * not incremented until after these activities are complete.
+ */
+void ipa_clock_get(struct ipa_clock *clock)
+{
+	/* If the clock is running, just bump the reference count */
+	if (ipa_clock_get_additional(clock))
+		return;
+
+	/* Otherwise get the mutex and check again */
+	mutex_lock(&clock->mutex);
+
+	/* A reference might have been added before we got the mutex. */
+	if (!ipa_clock_get_additional(clock)) {
+		int ret;
+
+		ret = ipa_clock_enable(clock);
+		if (!WARN(ret, "error %d enabling IPA clock\n", ret)) {
+			struct ipa *ipa = clock->ipa;
+
+			if (ipa->command_endpoint)
+				ipa_endpoint_resume(ipa->command_endpoint);
+
+			if (ipa->default_endpoint)
+				ipa_endpoint_resume(ipa->default_endpoint);
+
+			if (ipa->modem_netdev)
+				ipa_netdev_resume(ipa->modem_netdev);
+
+			atomic_inc(&clock->count);
+		}
+	}
+
+	mutex_unlock(&clock->mutex);
+}
+
+/* Attempt to remove an IPA clock reference.  If this represents
+ * the last reference, suspend endpoints and disable the clock
+ * (and interconnects) under protection of a mutex.
+ */
+void ipa_clock_put(struct ipa_clock *clock)
+{
+	/* If this is not the last reference there's nothing more to do */
+	if (!atomic_dec_and_mutex_lock(&clock->count, &clock->mutex))
+		return;
+
+	if (clock->ipa->modem_netdev)
+		ipa_netdev_suspend(clock->ipa->modem_netdev);
+
+	if (clock->ipa->default_endpoint)
+		ipa_endpoint_suspend(clock->ipa->default_endpoint);
+
+	if (clock->ipa->command_endpoint)
+		ipa_endpoint_suspend(clock->ipa->command_endpoint);
+
+	ipa_clock_disable(clock);
+
+	mutex_unlock(&clock->mutex);
+}
+
+/* Initialize IPA clocking */
+struct ipa_clock *ipa_clock_init(struct ipa *ipa)
+{
+	struct device *dev = &ipa->pdev->dev;
+	struct ipa_clock *clock;
+	int ret;
+
+	clock = kzalloc(sizeof(*clock), GFP_KERNEL);
+	if (!clock)
+		return ERR_PTR(-ENOMEM);
+
+	clock->ipa = ipa;
+	clock->core = clk_get(dev, "core");
+	if (IS_ERR(clock->core)) {
+		ret = PTR_ERR(clock->core);
+		goto err_free_clock;
+	}
+
+	ret = clk_set_rate(clock->core, IPA_CORE_CLOCK_RATE);
+	if (ret)
+		goto err_clk_put;
+
+	ret = ipa_interconnect_init(clock, dev);
+	if (ret)
+		goto err_clk_put;
+
+	mutex_init(&clock->mutex);
+	atomic_set(&clock->count, 0);
+
+	return clock;
+
+err_clk_put:
+	clk_put(clock->core);
+err_free_clock:
+	kfree(clock);
+
+	return ERR_PTR(ret);
+}
+
+/* Inverse of ipa_clock_init() */
+void ipa_clock_exit(struct ipa_clock *clock)
+{
+	mutex_destroy(&clock->mutex);
+	ipa_interconnect_exit(clock);
+	clk_put(clock->core);
+	kfree(clock);
+}
drivers/net/ipa/ipa_clock.h new file +52 -0

52 lines not on this page · raw message

drivers/net/ipa/ipa_interrupt.c new file +279 -0

279 lines not on this page · raw message

drivers/net/ipa/ipa_interrupt.h new file +53 -0

53 lines not on this page · raw message

drivers/net/ipa/ipa_mem.c new file +234 -0

234 lines not on this page · raw message

drivers/net/ipa/ipa_mem.h new file +83 -0

83 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#11 lore raw

PATCH v2 6/17 soc: qcom: ipa: GSI headers

In reply to Alex Elder

The Generic Software Interface is a layer of the IPA driver that abstracts the underlying hardware. The next patch includes the main code for GSI (including some additional documentation). This patch just includes three GSI header files.

- "gsi.h" is the top-level GSI header file. There is one of these associated with the IPA structure; in fact, it is embedded within the IPA structure. (Were it not embedded this way, many of the definitions structures defined here could be private to GSI code.) The main abstraction implemented by the GSI code is the channel, and this header exposes several operations that can be performed on a GSI channel.

- "gsi_private.h" exposes some definitions that are intended to be private, used only by the main GSI code and the GSI transaction code (defined in an upcoming patch).

- Like "ipa_reg.h", "gsi_reg.h" defines the offsets of the 32-bit registers used by the GSI layer, along with masks that define the position and width of fields less than 32 bits located within these registers.

---
 drivers/net/ipa/gsi.h         | 246 ++++++++++++++++++++++
 drivers/net/ipa/gsi_private.h | 148 +++++++++++++
 drivers/net/ipa/gsi_reg.h     | 376 ++++++++++++++++++++++++++++++++++
 3 files changed, 770 insertions(+)
 create mode 100644 drivers/net/ipa/gsi.h
 create mode 100644 drivers/net/ipa/gsi_private.h
 create mode 100644 drivers/net/ipa/gsi_reg.h
drivers/net/ipa/gsi.h new file +246 -0
new file mode 100644
index 000000000000..872ca682853a
@@ -0,0 +1,246 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Copyright (c) 2015-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+#ifndef _GSI_H_
+#define _GSI_H_
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+#include <linux/mutex.h>
+#include <linux/completion.h>
+#include <linux/platform_device.h>
+#include <linux/netdevice.h>
+
+#define GSI_CHANNEL_MAX		14
+#define GSI_EVT_RING_MAX	10
+
+struct device;
+struct scatterlist;
+struct platform_device;
+
+struct gsi;
+struct gsi_trans;
+struct gsi_channel_data;
+struct gsi_ipa_endpoint_data;
+
+/* Execution environment IDs */
+enum gsi_ee_id {
+	GSI_EE_AP	= 0,
+	GSI_EE_MODEM	= 1,
+	GSI_EE_UC	= 2,
+	GSI_EE_TZ	= 3,
+};
+
+/* Channel operation statistics, aggregated across all channels */
+struct gsi_channel_stats {
+	u64 allocate;
+	u64 start;
+	u64 stop;
+	u64 reset;
+	u64 free;
+};
+
+struct gsi_ring {
+	void *virt;			/* ring array base address */
+	dma_addr_t addr;		/* primarily low 32 bits used */
+	u32 count;			/* number of elements in ring */
+
+	/* The ring index value indicates the next "open" entry in the ring.
+	 *
+	 * A channel ring consists of TRE entries filled by the AP and passed
+	 * to the hardware for processing.  For a channel ring, the ring index
+	 * identifies the next unused entry to be filled by the AP.
+	 *
+	 * An event ring consists of event structures filled by the hardware
+	 * and passed to the AP.  For event rings, the ring index identifies
+	 * the next ring entry that is not known to have been filled by the
+	 * hardware.
+	 */
+	u32 index;
+};
+
+struct gsi_trans_info {
+	struct gsi_trans **map;		/* TRE -> transaction map */
+	u32 pool_count;			/* # transactions in the pool */
+	struct gsi_trans *pool;		/* transaction allocation pool */
+	u32 pool_free;			/* next free trans in pool (modulo) */
+	u32 sg_pool_count;		/* # SGs in the allocation pool */
+	struct scatterlist *sg_pool;	/* SG allocation pool */
+	u32 sg_pool_free;		/* next free SG pool entry */
+
+	atomic_t tre_avail;		/* # unallocated TREs in ring */
+	spinlock_t spinlock;		/* protects updates to the lists */
+	struct list_head alloc;		/* allocated, not committed */
+	struct list_head pending;	/* committed, awaiting completion */
+	struct list_head complete;	/* completed, awaiting poll */
+	struct list_head polled;	/* returned by gsi_channel_poll_one() */
+};
+
+/* Hardware values signifying the state of a channel */
+enum gsi_channel_state {
+	GSI_CHANNEL_STATE_NOT_ALLOCATED	= 0x0,
+	GSI_CHANNEL_STATE_ALLOCATED	= 0x1,
+	GSI_CHANNEL_STATE_STARTED	= 0x2,
+	GSI_CHANNEL_STATE_STOPPED	= 0x3,
+	GSI_CHANNEL_STATE_STOP_IN_PROC	= 0x4,
+	GSI_CHANNEL_STATE_ERROR		= 0xf,
+};
+
+/* We only care about channels between IPA and AP */
+struct gsi_channel {
+	struct gsi *gsi;
+	u32 toward_ipa;			/* 0: IPA->AP; 1: AP->IPA */
+
+	const struct gsi_channel_data *data;	/* initialization data */
+
+	struct completion completion;	/* signals channel state changes */
+	enum gsi_channel_state state;
+
+	struct gsi_ring tre_ring;
+	u32 evt_ring_id;
+
+	u64 byte_count;			/* total # bytes transferred */
+	u64 trans_count;		/* total # transactions */
+	/* The following counts are used only for TX endpoints */
+	u64 queued_byte_count;		/* last reported queued byte count */
+	u64 queued_trans_count;		/* ...and queued trans count */
+	u64 compl_byte_count;		/* last reported completed byte count */
+	u64 compl_trans_count;		/* ...and completed trans count */
+
+	struct gsi_trans_info trans_info;
+
+	struct napi_struct napi;
+};
+
+/* Hardware values signifying the state of an event ring */
+enum gsi_evt_ring_state {
+	GSI_EVT_RING_STATE_NOT_ALLOCATED	= 0x0,
+	GSI_EVT_RING_STATE_ALLOCATED		= 0x1,
+	GSI_EVT_RING_STATE_ERROR		= 0xf,
+};
+
+struct gsi_evt_ring {
+	struct gsi_channel *channel;
+	struct completion completion;	/* signals event ring state changes */
+	enum gsi_evt_ring_state state;
+	struct gsi_ring ring;
+};
+
+struct gsi {
+	struct device *dev;		/* Same as IPA device */
+	struct net_device dummy_dev;	/* needed for NAPI */
+	void __iomem *virt;
+	u32 irq;
+	u32 irq_wake_enabled;		/* 1: irq wake was enabled */
+	struct gsi_channel channel[GSI_CHANNEL_MAX];
+	struct gsi_channel_stats channel_stats;
+	struct gsi_evt_ring evt_ring[GSI_EVT_RING_MAX];
+	u32 event_bitmap;
+	u32 event_enable_bitmap;
+	struct mutex mutex;		/* protects commands, programming */
+};
+
+/**
+ * gsi_setup() - Set up the GSI subsystem
+ * @gsi:	Address of GSI structure embedded in an IPA structure
+ *
+ * @Return:	0 if successful, or a negative error code
+ *
+ * Performs initialization that must wait until the GSI hardware is
+ * ready (including firmware loaded).
+ */
+int gsi_setup(struct gsi *gsi);
+
+/**
+ * gsi_teardown() - Tear down GSI subsystem
+ * @gsi:	GSI address previously passed to a successful gsi_setup() call
+ */
+void gsi_teardown(struct gsi *gsi);
+
+/**
+ * gsi_channel_trans_max() - Channel maximum number of transactions
+ * @gsi:	GSI pointer
+ * @channel_id:	Channel whose limit is to be returned
+ *
+ * @Return:	 The maximum number of pending transactions on the channel
+ */
+u32 gsi_channel_trans_max(struct gsi *gsi, u32 channel_id);
+
+/**
+ * gsi_channel_trans_tre_max() - Return the maximum TREs per transaction
+ * @gsi:	GSI pointer
+ * @channel_id:	Channel whose limit is to be returned
+ *
+ * @Return:	 The maximum TRE count per transaction on the channel
+ */
+u32 gsi_channel_trans_tre_max(struct gsi *gsi, u32 channel_id);
+
+/**
+ * gsi_channel_trans_quiesce() - Wait for channel transactions to complete
+ * @gsi:	GSI pointer
+ * @channel_id:	Channel to quiesce
+ *
+ * Wait for all of a channel's currently-allocated transactions to
+ * be committed, complete, and be freed.
+ *
+ * NOTE:  Assumes no new transactions will be issued before it returns.
+ */
+void gsi_channel_trans_quiesce(struct gsi *gsi, u32 channel_id);
+
+/**
+ * gsi_channel_start() - Make a GSI channel operational
+ * @gsi:	GSI pointer
+ * @channel_id:	Channel to start
+ *
+ * @Return:	0 if successful, or a negative error code
+ */
+int gsi_channel_start(struct gsi *gsi, u32 channel_id);
+
+/**
+ * gsi_channel_stop() - Stop an operational GSI channel
+ * @gsi:	GSI pointer returned by gsi_setup()
+ * @channel_id:	Channel to stop
+ *
+ * @Return:	0 if successful, or a negative error code
+ */
+int gsi_channel_stop(struct gsi *gsi, u32 channel_id);
+
+/**
+ * gsi_channel_reset() - Reset a GSI channel
+ * @gsi:	GSI pointer
+ * @channel_id:	Channel to be reset
+ * @db_enable:	Whether doorbell engine should be enabled
+ *
+ * @Return:	0 if successful, or a negative error code
+ *
+ * Reset a channel and reconfigure it.  The @db_enable flag indicates
+ * whether the doorbell engine will be enabled following reconfiguration.
+ *
+ * GSI hardware relinquishes ownership of all pending receive buffer
+ * transactions as a result of reset.  They will be completed with
+ * result code -ECANCELED.
+ */
+int gsi_channel_reset(struct gsi *gsi, u32 channel_id, bool db_enable);
+
+/**
+ * gsi_init() - Initialize the GSI subsystem
+ * @gsi:	Address of GSI structure embedded in an IPA structure
+ * @pdev:	IPA platform device
+ *
+ * @Return:	0 if successful, or a negative error code
+ *
+ * Early stage initialization of the GSI subsystem, performing tasks
+ * that can be done before the GSI hardware is ready to use.
+ */
+int gsi_init(struct gsi *gsi, struct platform_device *pdev, u32 data_count,
+	     const struct gsi_ipa_endpoint_data *data);
+
+/**
+ * gsi_exit() - Exit the GSI subsystem
+ * @gsi:	GSI address previously passed to a successful gsi_init() call
+ */
+void gsi_exit(struct gsi *gsi);
+
+#endif /* _GSI_H_ */
drivers/net/ipa/gsi_private.h new file +148 -0

148 lines not on this page · raw message

drivers/net/ipa/gsi_reg.h new file +376 -0

376 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#12 lore raw

PATCH v2 7/17 soc: qcom: ipa: the generic software interface

In reply to Alex Elder

This patch includes "gsi.c", which implements the generic software interface (GSI) for IPA. The generic software interface abstracts channels, which provide a means of transferring data either from the AP to the IPA, or from the IPA to the AP. A ring buffer of "transfer elements" (TREs) is used to describe data transfers to perform. The AP writes a doorbell register associated with a channel to let it know it has added new entries (for an AP->IPA channel) or has finished processing entries (for an IPA->AP channel).

Each channel also has an event ring buffer, used by the IPA to communicate information about events related to a channel (for example, the completion of TREs). The IPA writes its own doorbell register, which triggers an interrupt on the AP, to signal that new event information has arrived.

---
 drivers/net/ipa/gsi.c | 1635 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 1635 insertions(+)
 create mode 100644 drivers/net/ipa/gsi.c
drivers/net/ipa/gsi.c new file +1635 -0
new file mode 100644
index 000000000000..a749d3b0d792
@@ -0,0 +1,1635 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2015-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/bits.h>
+#include <linux/bitfield.h>
+#include <linux/mutex.h>
+#include <linux/completion.h>
+#include <linux/io.h>
+#include <linux/bug.h>
+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/netdevice.h>
+
+#include "gsi.h"
+#include "gsi_reg.h"
+#include "gsi_private.h"
+#include "gsi_trans.h"
+#include "ipa_gsi.h"
+#include "ipa_data.h"
+
+/**
+ * DOC: The IPA Generic Software Interface
+ *
+ * The generic software interface (GSI) is an integral component of the IPA,
+ * providing a well-defined communication layer between the AP subsystem
+ * and the IPA core.  The modem uses the GSI layer as well.
+ *
+ *	--------	     ---------
+ *	|      |	     |	     |
+ *	|  AP  +<---.	.----+ Modem |
+ *	|      +--. |	| .->+	     |
+ *	|      |  | |	| |  |	     |
+ *	--------  | |	| |  ---------
+ *		  v |	v |
+ *		--+-+---+-+--
+ *		|    GSI    |
+ *		|-----------|
+ *		|	    |
+ *		|    IPA    |
+ *		|	    |
+ *		-------------
+ *
+ * In the above diagram, the AP and Modem represent "execution environments"
+ * (EEs), which are independent operating environments that use the IPA for
+ * data transfer.
+ *
+ * Each EE uses a set of unidirectional GSI "channels," which allow transfer
+ * of data to or from the IPA.  A channel is implemented as a ring buffer,
+ * with a DRAM-resident array of "transfer elements" (TREs) available to
+ * describe transfers to or from other EEs through the IPA.  A transfer
+ * element can also contain an immediate command, requesting the IPA perform
+ * actions other than data transfer.
+ *
+ * Each TRE refers to a block of data--also located DRAM.  After writing one
+ * or more TREs to a channel, the writer (either the IPA or an EE) writes a
+ * doorbell register to inform the receiving side how many elements have
+ * been written.  Writing to a doorbell register triggers within the GSI.
+ *
+ * Each channel has a GSI "event ring" associated with it.  An event ring
+ * is implemented very much like a channel ring, but is always directed from
+ * the IPA to an EE.  The IPA notifies an EE (such as the AP) about channel
+ * events by adding an entry to the event ring associated with the channel.
+ * The GSI then writes its doorbell for the event ring, causing the target
+ * EE to be interrupted.  Each entry in an event ring contains a pointer
+ * to the channel TRE whose completion the event represents.
+ *
+ * Each TRE in a channel ring has a set of flags.  One flag indicates whether
+ * the completion of the transfer operation generates an entry (and possibly
+ * an interrupt) in the channel's event ring.  Other flags allow transfer
+ * elements to be chained together, forming a single logical transaction.
+ * TRE flags are used to control whether and when interrupts are generated
+ * to signal completion of channel transfers.
+ *
+ * Elements in channel and event rings are completed (or consumed) strictly
+ * in order.  Completion of one entry implies the completion of all preceding
+ * entries.  A single completion interrupt can therefore communicate the
+ * completion of many transfers.
+ *
+ * Note that all GSI registers are little-endian, which is the assumed
+ * endianness of I/O space accesses.  The accessor functions perform byte
+ * swapping if needed (i.e., for a big endian CPU).
+ */
+
+/* Delay period for interrupt moderation (in 32KHz IPA internal timer ticks) */
+#define IPA_GSI_EVT_RING_INT_MODT	(32 * 1) /* 1ms under 32KHz clock */
+
+#define GSI_CMD_TIMEOUT		5	/* seconds */
+
+#define GSI_MHI_ER_START	10	/* First reserved event number */
+#define GSI_MHI_ER_END		16	/* Last reserved event number */
+
+#define GSI_ISR_MAX_ITER	50	/* Detect interrupt storms */
+
+/* Hardware values from the error log register error code field */
+enum gsi_err_code {
+	GSI_INVALID_TRE_ERR			= 0x1,
+	GSI_OUT_OF_BUFFERS_ERR			= 0x2,
+	GSI_OUT_OF_RESOURCES_ERR		= 0x3,
+	GSI_UNSUPPORTED_INTER_EE_OP_ERR		= 0x4,
+	GSI_EVT_RING_EMPTY_ERR			= 0x5,
+	GSI_NON_ALLOCATED_EVT_ACCESS_ERR	= 0x6,
+	GSI_HWO_1_ERR				= 0x8,
+};
+
+/* Hardware values from the error log register error type field */
+enum gsi_err_type {
+	GSI_ERR_TYPE_GLOB	= 0x1,
+	GSI_ERR_TYPE_CHAN	= 0x2,
+	GSI_ERR_TYPE_EVT	= 0x3,
+};
+
+/* Fields in an error log register at GSI_ERROR_LOG_OFFSET */
+#define GSI_LOG_ERR_ARG3_FMASK		GENMASK(3, 0)
+#define GSI_LOG_ERR_ARG2_FMASK		GENMASK(7, 4)
+#define GSI_LOG_ERR_ARG1_FMASK		GENMASK(11, 8)
+#define GSI_LOG_ERR_CODE_FMASK		GENMASK(15, 12)
+#define GSI_LOG_ERR_VIRT_IDX_FMASK	GENMASK(23, 19)
+#define GSI_LOG_ERR_TYPE_FMASK		GENMASK(27, 24)
+#define GSI_LOG_ERR_EE_FMASK		GENMASK(31, 28)
+
+/* Hardware values used when programming an event ring */
+enum gsi_evt_chtype {
+	GSI_EVT_CHTYPE_MHI_EV	= 0x0,
+	GSI_EVT_CHTYPE_XHCI_EV	= 0x1,
+	GSI_EVT_CHTYPE_GPI_EV	= 0x2,
+	GSI_EVT_CHTYPE_XDCI_EV	= 0x3,
+};
+
+/* Hardware values used when programming a channel */
+enum gsi_channel_protocol {
+	GSI_CHANNEL_PROTOCOL_MHI	= 0x0,
+	GSI_CHANNEL_PROTOCOL_XHCI	= 0x1,
+	GSI_CHANNEL_PROTOCOL_GPI	= 0x2,
+	GSI_CHANNEL_PROTOCOL_XDCI	= 0x3,
+};
+
+/* Hardware values representing an event ring immediate command opcode */
+enum gsi_evt_ch_cmd_opcode {
+	GSI_EVT_ALLOCATE	= 0x0,
+	GSI_EVT_RESET		= 0x9,
+	GSI_EVT_DE_ALLOC	= 0xa,
+};
+
+/* Hardware values representing a channel immediate command opcode */
+enum gsi_ch_cmd_opcode {
+	GSI_CH_ALLOCATE	= 0x0,
+	GSI_CH_START	= 0x1,
+	GSI_CH_STOP	= 0x2,
+	GSI_CH_RESET	= 0x9,
+	GSI_CH_DE_ALLOC	= 0xa,
+	GSI_CH_DB_STOP	= 0xb,
+};
+
+/** gsi_gpi_channel_scratch - GPI protocol scratch register
+ *
+ * @max_outstanding_tre:
+ *	Defines the maximum number of TREs allowed in a single transaction
+ *	on a channel (in Bytes).  This determines the amount of prefetch
+ *	performed by the hardware.  We configure this to equal the size of
+ *	the TLV FIFO for the channel.
+ * @outstanding_threshold:
+ *	Defines the threshold (in Bytes) determining when the sequencer
+ *	should update the channel doorbell.  We configure this to equal
+ *	the size of two TREs.
+ */
+struct gsi_gpi_channel_scratch {
+	u64 reserved1;
+	u16 reserved2;
+	u16 max_outstanding_tre;
+	u16 reserved3;
+	u16 outstanding_threshold;
+};
+
+/** gsi_channel_scratch - channel scratch configuration area
+ *
+ * The exact interpretation of this register is protocol-specific.
+ * We only use GPI channels; see struct gsi_gpi_channel_scratch, above.
+ */
+union gsi_channel_scratch {
+	struct gsi_gpi_channel_scratch gpi;
+	struct {
+		u32 word1;
+		u32 word2;
+		u32 word3;
+		u32 word4;
+	} data;
+};
+
+/* Return the channel id associated with a given channel */
+static u32 gsi_channel_id(struct gsi_channel *channel)
+{
+	return channel - &channel->gsi->channel[0];
+}
+
+/* Report the number of bytes queued to hardware since last call */
+void gsi_channel_tx_queued(struct gsi_channel *channel)
+{
+	u32 trans_count;
+	u32 byte_count;
+
+	trans_count = channel->trans_count - channel->queued_trans_count;
+	byte_count = channel->byte_count - channel->queued_byte_count;
+	channel->queued_trans_count = channel->trans_count;
+	channel->queued_byte_count = channel->byte_count;
+
+	ipa_gsi_channel_tx_queued(channel->gsi, gsi_channel_id(channel),
+				  trans_count, byte_count);
+}
+
+static void gsi_irq_event_enable(struct gsi *gsi, u32 evt_ring_id)
+{
+	u32 val;
+
+	gsi->event_enable_bitmap |= BIT(evt_ring_id);
+	val = gsi->event_enable_bitmap;
+	iowrite32(val, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_MSK_OFFSET);
+}
+
+static void gsi_irq_event_disable(struct gsi *gsi, u32 evt_ring_id)
+{
+	u32 val;
+
+	gsi->event_enable_bitmap &= ~BIT(evt_ring_id);
+	val = gsi->event_enable_bitmap;
+	iowrite32(val, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_MSK_OFFSET);
+}
+
+/* Enable all GSI_interrupt types */
+static void gsi_irq_enable(struct gsi *gsi)
+{
+	u32 val;
+
+	/* Inter EE commands / interrupt are not supported. */
+	val = GSI_CNTXT_TYPE_IRQ_MSK_ALL;
+	iowrite32(val, gsi->virt + GSI_CNTXT_TYPE_IRQ_MSK_OFFSET);
+
+	val = GENMASK(GSI_CHANNEL_MAX - 1, 0);
+	iowrite32(val, gsi->virt + GSI_CNTXT_SRC_CH_IRQ_MSK_OFFSET);
+
+	val = GENMASK(GSI_EVT_RING_MAX - 1, 0);
+	iowrite32(val, gsi->virt + GSI_CNTXT_SRC_EV_CH_IRQ_MSK_OFFSET);
+
+	/* Each IEOB interrupt is enabled (later) as needed by channels */
+	iowrite32(0, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_MSK_OFFSET);
+
+	val = GSI_CNTXT_GLOB_IRQ_ALL;
+	iowrite32(val, gsi->virt + GSI_CNTXT_GLOB_IRQ_EN_OFFSET);
+
+	/* Never enable GSI_BREAK_POINT */
+	val = GSI_CNTXT_GSI_IRQ_ALL & ~EN_BREAK_POINT_FMASK;
+	iowrite32(val, gsi->virt + GSI_CNTXT_GSI_IRQ_EN_OFFSET);
+}
+
+/* Disable all GSI_interrupt types */
+static void gsi_irq_disable(struct gsi *gsi)
+{
+	iowrite32(0, gsi->virt + GSI_CNTXT_GSI_IRQ_EN_OFFSET);
+	iowrite32(0, gsi->virt + GSI_CNTXT_GLOB_IRQ_EN_OFFSET);
+	iowrite32(0, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_MSK_OFFSET);
+	iowrite32(0, gsi->virt + GSI_CNTXT_SRC_EV_CH_IRQ_MSK_OFFSET);
+	iowrite32(0, gsi->virt + GSI_CNTXT_SRC_CH_IRQ_MSK_OFFSET);
+	iowrite32(0, gsi->virt + GSI_CNTXT_TYPE_IRQ_MSK_OFFSET);
+}
+
+/* Return the hardware's notion of the current state of a channel */
+static enum gsi_channel_state gsi_channel_state(struct gsi_channel *channel)
+{
+	u32 channel_id = gsi_channel_id(channel);
+	struct gsi *gsi = channel->gsi;
+	u32 val;
+
+	val = ioread32(gsi->virt + GSI_CH_C_CNTXT_0_OFFSET(channel_id));
+
+	return u32_get_bits(val, CHSTATE_FMASK);
+}
+
+/* Return the hardware's notion of the current state of an event ring */
+static enum gsi_evt_ring_state
+gsi_evt_ring_state(struct gsi *gsi, u32 evt_ring_id)
+{
+	u32 val = ioread32(gsi->virt + GSI_EV_CH_E_CNTXT_0_OFFSET(evt_ring_id));
+
+	return u32_get_bits(val, EV_CHSTATE_FMASK);
+}
+
+/* Channel control interrupt handler */
+static void gsi_isr_chan_ctrl(struct gsi *gsi)
+{
+	u32 channel_mask;
+
+	channel_mask = ioread32(gsi->virt + GSI_CNTXT_SRC_CH_IRQ_OFFSET);
+	iowrite32(channel_mask, gsi->virt + GSI_CNTXT_SRC_CH_IRQ_CLR_OFFSET);
+
+	while (channel_mask) {
+		u32 channel_id = __ffs(channel_mask);
+		struct gsi_channel *channel;
+
+		channel_mask ^= BIT(channel_id);
+
+		channel = &gsi->channel[channel_id];
+		channel->state = gsi_channel_state(channel);
+
+		complete(&channel->completion);
+	}
+}
+
+static void gsi_isr_evt_ctrl(struct gsi *gsi)
+{
+	u32 event_mask;
+
+	event_mask = ioread32(gsi->virt + GSI_CNTXT_SRC_EV_CH_IRQ_OFFSET);
+	iowrite32(event_mask, gsi->virt + GSI_CNTXT_SRC_EV_CH_IRQ_CLR_OFFSET);
+
+	while (event_mask) {
+		u32 evt_ring_id = __ffs(event_mask);
+		struct gsi_evt_ring *evt_ring;
+
+		event_mask ^= BIT(evt_ring_id);
+
+		evt_ring = &gsi->evt_ring[evt_ring_id];
+		evt_ring->state = gsi_evt_ring_state(gsi, evt_ring_id);
+
+		complete(&evt_ring->completion);
+	}
+}
+
+static void
+gsi_isr_glob_chan_err(struct gsi *gsi, u32 err_ee, u32 channel_id, u32 code)
+{
+	if (code == GSI_OUT_OF_RESOURCES_ERR) {
+		dev_err(gsi->dev, "channel %u out of resources\n", channel_id);
+		complete(&gsi->channel[channel_id].completion);
+		return;
+	}
+
+	/* Report, but otherwise ignore all other error codes */
+	dev_err(gsi->dev, "channel %u global error ee 0x%08x code 0x%08x\n",
+		channel_id, err_ee, code);
+}
+
+static void
+gsi_isr_glob_evt_err(struct gsi *gsi, u32 err_ee, u32 evt_ring_id, u32 code)
+{
+	if (code == GSI_OUT_OF_RESOURCES_ERR) {
+		struct gsi_evt_ring *evt_ring = &gsi->evt_ring[evt_ring_id];
+		u32 channel_id = gsi_channel_id(evt_ring->channel);
+
+		complete(&evt_ring->completion);
+		dev_err(gsi->dev, "evt_ring for channel %u out of resources\n",
+			channel_id);
+		return;
+	}
+
+	/* Report, but otherwise ignore all other error codes */
+	dev_err(gsi->dev, "event ring %u global error ee %u code 0x%08x\n",
+		evt_ring_id, err_ee, code);
+}
+
+static void gsi_isr_glob_err(struct gsi *gsi)
+{
+	enum gsi_err_type type;
+	enum gsi_err_code code;
+	u32 which;
+	u32 val;
+	u32 ee;
+
+	/* Get the logged error, then reinitialize the log */
+	val = ioread32(gsi->virt + GSI_ERROR_LOG_OFFSET);
+	iowrite32(0, gsi->virt + GSI_ERROR_LOG_OFFSET);
+	iowrite32(~0, gsi->virt + GSI_ERROR_LOG_CLR_OFFSET);
+
+	ee = u32_get_bits(val, GSI_LOG_ERR_EE_FMASK);
+	which = u32_get_bits(val, GSI_LOG_ERR_VIRT_IDX_FMASK);
+	type = u32_get_bits(val, GSI_LOG_ERR_TYPE_FMASK);
+	code = u32_get_bits(val, GSI_LOG_ERR_CODE_FMASK);
+
+	if (type == GSI_ERR_TYPE_CHAN)
+		gsi_isr_glob_chan_err(gsi, ee, which, code);
+	else if (type == GSI_ERR_TYPE_EVT)
+		gsi_isr_glob_evt_err(gsi, ee, which, code);
+	else	/* type GSI_ERR_TYPE_GLOB should be fatal */
+		dev_err(gsi->dev, "unexpected global error 0x%08x\n", type);
+}
+
+static void gsi_isr_glob_ee(struct gsi *gsi)
+{
+	u32 val;
+
+	val = ioread32(gsi->virt + GSI_CNTXT_GLOB_IRQ_STTS_OFFSET);
+
+	if (val & ERROR_INT_FMASK)
+		gsi_isr_glob_err(gsi);
+
+	iowrite32(val, gsi->virt + GSI_CNTXT_GLOB_IRQ_CLR_OFFSET);
+
+	val &= ~ERROR_INT_FMASK;
+
+	if (val & EN_GP_INT1_FMASK) {
+		dev_err(gsi->dev, "unexpected global INT1\n");
+		val ^= EN_GP_INT1_FMASK;
+	}
+
+	if (val)
+		dev_err(gsi->dev, "unexpected global interrupt 0x%08x\n", val);
+}
+
+/* I/O completion interrupt event */
+static void gsi_isr_ieob(struct gsi *gsi)
+{
+	u32 event_mask;
+
+	event_mask = ioread32(gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_OFFSET);
+	iowrite32(event_mask, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_CLR_OFFSET);
+
+	while (event_mask) {
+		u32 evt_ring_id = __ffs(event_mask);
+
+		event_mask ^= BIT(evt_ring_id);
+
+		gsi_irq_event_disable(gsi, evt_ring_id);
+		napi_schedule(&gsi->evt_ring[evt_ring_id].channel->napi);
+	}
+}
+
+/* We don't currently expect to receive any inter-EE channel interrupts */
+static void gsi_isr_inter_ee_chan_ctrl(struct gsi *gsi)
+{
+	u32 channel_mask;
+
+	channel_mask = ioread32(gsi->virt + GSI_INTER_EE_SRC_CH_IRQ_OFFSET);
+	iowrite32(channel_mask, gsi->virt + GSI_INTER_EE_SRC_CH_IRQ_CLR_OFFSET);
+
+	while (channel_mask) {
+		u32 channel_id = __ffs(channel_mask);
+
+		dev_err(gsi->dev, "ch %u inter-EE interrupt\n", channel_id);
+		channel_mask ^= BIT(channel_id);
+	}
+}
+
+/* We don't currently expect to receive any inter-EE event interrupts */
+static void gsi_isr_inter_ee_evt_ctrl(struct gsi *gsi)
+{
+	u32 event_mask;
+
+	event_mask = ioread32(gsi->virt + GSI_INTER_EE_SRC_EV_CH_IRQ_OFFSET);
+	iowrite32(event_mask,
+		  gsi->virt + GSI_INTER_EE_SRC_EV_CH_IRQ_CLR_OFFSET);
+
+	while (event_mask) {
+		u32 evt_ring_id = __ffs(event_mask);
+
+		event_mask ^= BIT(evt_ring_id);
+
+		/* not currently expected */
+		dev_err(gsi->dev, "evt %u inter-EE interrupt\n", evt_ring_id);
+	}
+}
+
+/* We don't currently expect to receive any general event interrupts */
+static void gsi_isr_general(struct gsi *gsi)
+{
+	u32 val;
+
+	val = ioread32(gsi->virt + GSI_CNTXT_GSI_IRQ_STTS_OFFSET);
+	iowrite32(val, gsi->virt + GSI_CNTXT_GSI_IRQ_CLR_OFFSET);
+
+	if (val & CLR_BREAK_POINT_FMASK)
+		dev_err(gsi->dev, "breakpoint!\n");
+	val ^= CLR_BREAK_POINT_FMASK;
+
+	if (val)
+		dev_err(gsi->dev, "unexpected general interrupt 0x%08x\n", val);
+}
+
+/**
+ * gsi_isr() - Top level GSI interrupt service routine
+ * @irq:	Interrupt number (ignored)
+ * @dev_id:	GSI pointer supplied to request_irq()
+ *
+ * This is the main handler function registered for the GSI IRQ. Each type
+ * of interrupt has a separate handler function that is called from here.
+ */
+static irqreturn_t gsi_isr(int irq, void *dev_id)
+{
+	struct gsi *gsi = dev_id;
+	u32 intr_mask;
+	u32 cnt = 0;
+
+	while ((intr_mask = ioread32(gsi->virt + GSI_CNTXT_TYPE_IRQ_OFFSET))) {
+		/* intr_mask contains bitmask of pending GSI interrupts */
+		do {
+			u32 gsi_intr = BIT(__ffs(intr_mask));
+
+			intr_mask ^= gsi_intr;
+
+			switch (gsi_intr) {
+			case CH_CTRL_FMASK:
+				gsi_isr_chan_ctrl(gsi);
+				break;
+			case EV_CTRL_FMASK:
+				gsi_isr_evt_ctrl(gsi);
+				break;
+			case GLOB_EE_FMASK:
+				gsi_isr_glob_ee(gsi);
+				break;
+			case IEOB_FMASK:
+				gsi_isr_ieob(gsi);
+				break;
+			case INTER_EE_CH_CTRL_FMASK:
+				gsi_isr_inter_ee_chan_ctrl(gsi);
+				break;
+			case INTER_EE_EV_CTRL_FMASK:
+				gsi_isr_inter_ee_evt_ctrl(gsi);
+				break;
+			case GENERAL_FMASK:
+				gsi_isr_general(gsi);
+				break;
+			default:
+				dev_err(gsi->dev,
+					"%s: unrecognized type 0x%08x\n",
+					__func__, gsi_intr);
+				break;
+			}
+		} while (intr_mask);
+
+		if (++cnt > GSI_ISR_MAX_ITER) {
+			dev_err(gsi->dev, "interrupt flood\n");
+			break;
+		}
+	}
+
+	return IRQ_HANDLED;
+}
+
+/* Return the virtual address associated with a ring index */
+void *gsi_ring_virt(struct gsi_ring *ring, u32 index)
+{
+	/* Note: index *must* be used modulo the ring count here */
+	return ring->virt + (index % ring->count) * sizeof(struct gsi_tre);
+}
+
+/* Return the 32-bit DMA address associated with a ring index */
+u32 gsi_ring_addr(struct gsi_ring *ring, u32 index)
+{
+	return (ring->addr & GENMASK(31, 0)) + index * sizeof(struct gsi_tre);
+}
+
+/* Return the ring index of a 32-bit ring offset */
+static u32 gsi_ring_index(struct gsi_ring *ring, u32 offset)
+{
+	/* Code assumes channel and event ring elements are the same size */
+	BUILD_BUG_ON(sizeof(struct gsi_tre) !=
+		     sizeof(struct gsi_xfer_compl_evt));
+
+	return (offset - gsi_ring_addr(ring, 0)) / sizeof(struct gsi_tre);
+}
+
+/* Return the transaction associated with a transfer completion event */
+static struct gsi_trans *gsi_event_trans(struct gsi_channel *channel,
+					 struct gsi_xfer_compl_evt *evt)
+{
+	u32 tre_offset;
+	u32 tre_index;
+
+	/* Event xfer_ptr records the TRE it's associated with */
+	tre_offset = le64_to_cpu(evt->xfer_ptr) & GENMASK(31, 0);
+	tre_index = gsi_ring_index(&channel->tre_ring, tre_offset);
+
+	return gsi_channel_trans_mapped(channel, tre_index);
+}
+/**
+ * gsi_channel_tx_update() - Report completed TX transfers
+ * @channel:	Channel that has completed transmitting packets
+ * @trans:	Last transation known to be complete
+ *
+ * Compute the number of transactions and bytes that have been
+ * transferred on a TX channel, and report that to higher layers in
+ * the network stack for throttling.
+ */
+static void
+gsi_channel_tx_update(struct gsi_channel *channel, struct gsi_trans *trans)
+{
+	u64 byte_count = trans->byte_count + trans->len;
+	u64 trans_count = trans->trans_count + 1;
+
+	byte_count -= channel->compl_byte_count;
+	channel->compl_byte_count += byte_count;
+	trans_count -= channel->compl_trans_count;
+	channel->compl_trans_count += trans_count;;
+	/* assert(trans_count <= U32_MAX); */
+
+	ipa_gsi_channel_tx_completed(channel->gsi, gsi_channel_id(channel),
+				     trans_count, byte_count);
+}
+
+/**
+ * gsi_evt_ring_rx_update() - Record lengths of received data
+ * @evt_ring:	Event ring associated with channel that received packets
+ * @index:	Event index in ring reported by hardware
+ *
+ * Events for RX channels contain the actual number of bytes received into
+ * the buffer.  Every event has a transaction associated with it, and here
+ * we update transactions to record their actual received lengths.
+ *
+ * This function is called whenever we learn that the GSI hardware has filled
+ * new events since the last time we checked.  The ring's index field tells
+ * the first entry in need of processing.  The index provided is the
+ * first *unfilled* event in the ring (following the last filled one).
+ *
+ * Events are sequential within the event ring, and transactions are
+ * sequential within the transaction pool.
+ *
+ * Note that @index always refers to an element *within* the event ring.
+ */
+static void gsi_evt_ring_rx_update(struct gsi_evt_ring *evt_ring, u32 index)
+{
+	struct gsi_channel *channel = evt_ring->channel;
+	struct gsi_ring *ring = &evt_ring->ring;
+	struct gsi_xfer_compl_evt *evt_done;
+	struct gsi_trans_info *trans_info;
+	struct gsi_xfer_compl_evt *evt;
+	struct gsi_trans *trans;
+	u32 byte_count = 0;
+	u32 trans_avail;
+	u32 old_index;
+	u32 evt_avail;
+
+	/* We'll start with the oldest un-processed event.  RX channels
+	 * replenish receive buffers in single-TRE transactions, so we
+	 * can just map that event to its transaction.
+	 */
+	old_index = ring->index;
+	evt = gsi_ring_virt(ring, old_index);
+	trans = gsi_event_trans(channel, evt);
+
+	/* Compute the number of events to process before we wrap */
+	evt_avail = ring->count - old_index % ring->count;
+
+	/* And compute how many transactions to process before we wrap */
+	trans_info = &channel->trans_info;
+	trans_avail = (u32)(&trans_info->pool[trans_info->pool_count] - trans);
+
+	/* Finally, determine when we'll be done processing events */
+	evt_done = gsi_ring_virt(ring, index);
+	do {
+		trans->len = __le16_to_cpu(evt->len);
+		byte_count += trans->result;
+
+		if (--evt_avail)
+			evt++;
+		else
+			evt = gsi_ring_virt(ring, 0);
+
+		if (--trans_avail)
+			trans++;
+		else
+			trans = &trans_info->pool[0];
+	} while (evt != evt_done);
+
+	/* We record RX bytes when they are received */
+	channel->byte_count += byte_count;
+	channel->trans_count++;
+}
+
+/* Ring an event ring doorbell, reporting the last entry processed by the AP.
+ * The index argument (modulo the ring count) is the first unfilled entry, so
+ * we supply one less than that with the doorbell.  Update the event ring
+ * index field with the value provided.
+ */
+static void gsi_evt_ring_doorbell(struct gsi *gsi, u32 evt_ring_id, u32 index)
+{
+	struct gsi_ring *ring = &gsi->evt_ring[evt_ring_id].ring;
+	u32 val;
+
+	ring->index = index;	/* Next unused entry */
+
+	/* Note: index *must* be used modulo the ring count here */
+	val = gsi_ring_addr(ring, (index - 1) % ring->count);
+	iowrite32(val, gsi->virt + GSI_EV_CH_E_DOORBELL_0_OFFSET(evt_ring_id));
+}
+
+/* Return the maximum number of channels the hardware supports */
+static u32 gsi_channel_max(struct gsi *gsi)
+{
+	u32 val = ioread32(gsi->virt + GSI_GSI_HW_PARAM_2_OFFSET);
+
+	return u32_get_bits(val, NUM_CH_PER_EE_FMASK);
+}
+
+/* Return the maximum number of event rings the hardware supports */
+static u32 gsi_evt_ring_max(struct gsi *gsi)
+{
+	u32 val = ioread32(gsi->virt + GSI_GSI_HW_PARAM_2_OFFSET);
+
+	return u32_get_bits(val, NUM_EV_PER_EE_FMASK);
+}
+
+/* Issue a GSI command by writing a value to a register, then wait for
+ * completion to be signaled.  Reports an error if the command times out.
+ * (Timeout is not expected, and suggests broken hardware.)
+ */
+static void
+gsi_command(struct gsi *gsi, u32 reg, u32 val, struct completion *completion)
+{
+	reinit_completion(completion);
+
+	iowrite32(val, gsi->virt + reg);
+	if (!wait_for_completion_timeout(completion, GSI_CMD_TIMEOUT * HZ))
+		dev_err(gsi->dev, "%s timeout reg 0x%08x val 0x%08x\n",
+			__func__, reg, val);
+}
+
+/* Issue an event ring command and wait for it to complete */
+static void evt_ring_command(struct gsi *gsi, u32 evt_ring_id,
+			     enum gsi_evt_ch_cmd_opcode op)
+{
+	struct completion *completion = &gsi->evt_ring[evt_ring_id].completion;
+	u32 val = 0;
+
+	val |= u32_encode_bits(evt_ring_id, EV_CHID_FMASK);
+	val |= u32_encode_bits(op, EV_OPCODE_FMASK);
+
+	gsi_command(gsi, GSI_EV_CH_CMD_OFFSET, val, completion);
+}
+
+/* Issue a channel command and wait for it to complete */
+static void
+gsi_channel_command(struct gsi_channel *channel, enum gsi_ch_cmd_opcode op)
+{
+	u32 channel_id = gsi_channel_id(channel);
+	u32 val = 0;
+
+	val |= u32_encode_bits(channel_id, CH_CHID_FMASK);
+	val |= u32_encode_bits(op, CH_OPCODE_FMASK);
+
+	gsi_command(channel->gsi, GSI_CH_CMD_OFFSET, val, &channel->completion);
+}
+
+/* Initialize a ring, including allocating DMA memory for its entries */
+static int gsi_ring_alloc(struct gsi *gsi, struct gsi_ring *ring, u32 count)
+{
+	size_t size = count * sizeof(struct gsi_tre);
+	dma_addr_t addr;
+
+	BUILD_BUG_ON(!is_power_of_2(sizeof(struct gsi_tre)));
+
+	if (!count)
+		return -EINVAL;
+
+	/* Hardware requires a 2^n ring size, with alignment equal to size */
+	ring->virt = dma_alloc_coherent(gsi->dev, size, &addr, GFP_KERNEL);
+	if (ring->virt && addr % size) {
+		dma_free_coherent(gsi->dev, size, ring->virt, ring->addr);
+		dev_err(gsi->dev, "unable to alloc 0x%zx-aligned ring buffer\n",
+				size);
+		return -EINVAL;	/* Not a good error value, but distinct */
+	} else if (!ring->virt) {
+		return -ENOMEM;
+	}
+	ring->addr = addr;
+	ring->count = count;
+
+	return 0;
+}
+
+/* Free a previously-allocated ring */
+static void gsi_ring_free(struct gsi *gsi, struct gsi_ring *ring)
+{
+	size_t size = ring->count * sizeof(struct gsi_tre);
+
+	dma_free_coherent(gsi->dev, size, ring->virt, ring->addr);
+}
+
+/* Program an event ring for use */
+static void gsi_evt_ring_program(struct gsi *gsi, u32 evt_ring_id)
+{
+	struct gsi_evt_ring *evt_ring = &gsi->evt_ring[evt_ring_id];
+	size_t size = evt_ring->ring.count * sizeof(struct gsi_tre);
+	u32 val = 0;
+
+	BUILD_BUG_ON(sizeof(struct gsi_xfer_compl_evt) >
+		     field_max(EV_ELEMENT_SIZE_FMASK));
+
+	val |= u32_encode_bits(GSI_EVT_CHTYPE_GPI_EV, EV_CHTYPE_FMASK);
+	val |= EV_INTYPE_FMASK;
+	val |= u32_encode_bits(sizeof(struct gsi_xfer_compl_evt),
+			       EV_ELEMENT_SIZE_FMASK);
+	iowrite32(val, gsi->virt + GSI_EV_CH_E_CNTXT_0_OFFSET(evt_ring_id));
+
+	val = u32_encode_bits(size, EV_R_LENGTH_FMASK);
+	iowrite32(val, gsi->virt + GSI_EV_CH_E_CNTXT_1_OFFSET(evt_ring_id));
+
+	/* The context 2 and 3 registers store the low-order and
+	 * high-order 32 bits of the address of the event ring,
+	 * respectively.
+	 */
+	val = evt_ring->ring.addr & GENMASK(31, 0);
+	iowrite32(val, gsi->virt + GSI_EV_CH_E_CNTXT_2_OFFSET(evt_ring_id));
+
+	val = evt_ring->ring.addr >> 32;
+	iowrite32(val, gsi->virt + GSI_EV_CH_E_CNTXT_3_OFFSET(evt_ring_id));
+
+	/* Enable interrupt moderation by setting the moderation delay */
+	val = u32_encode_bits(IPA_GSI_EVT_RING_INT_MODT, MODT_FMASK);
+	val |= u32_encode_bits(1, MODC_FMASK);	/* comes from channel */
+	iowrite32(val, gsi->virt + GSI_EV_CH_E_CNTXT_8_OFFSET(evt_ring_id));
+
+	/* No MSI write data, and MSI address high and low address is 0 */
+	iowrite32(0, gsi->virt + GSI_EV_CH_E_CNTXT_9_OFFSET(evt_ring_id));
+	iowrite32(0, gsi->virt + GSI_EV_CH_E_CNTXT_10_OFFSET(evt_ring_id));
+	iowrite32(0, gsi->virt + GSI_EV_CH_E_CNTXT_11_OFFSET(evt_ring_id));
+
+	/* We don't need to get event read pointer updates */
+	iowrite32(0, gsi->virt + GSI_EV_CH_E_CNTXT_12_OFFSET(evt_ring_id));
+	iowrite32(0, gsi->virt + GSI_EV_CH_E_CNTXT_13_OFFSET(evt_ring_id));
+}
+
+/* Issue an allocation request to the hardware for an event ring */
+static int gsi_evt_ring_alloc_hw(struct gsi *gsi, u32 evt_ring_id)
+{
+	struct gsi_evt_ring *evt_ring = &gsi->evt_ring[evt_ring_id];
+
+	evt_ring_command(gsi, evt_ring_id, GSI_EVT_ALLOCATE);
+
+	if (evt_ring->state != GSI_EVT_RING_STATE_ALLOCATED) {
+		dev_err(gsi->dev, "evt_ring_id %u allocation bad state %u\n",
+			evt_ring_id, evt_ring->state);
+		return -EIO;
+	}
+
+	gsi_evt_ring_program(gsi, evt_ring_id);
+
+	/* Have the first event in the ring be the first one filled. */
+	gsi_evt_ring_doorbell(gsi, evt_ring_id, 0);
+
+	return 0;
+}
+
+/* Issue a hardware de-allocation request for an (allocated) event ring */
+static void gsi_evt_ring_free_hw(struct gsi *gsi, u32 evt_ring_id)
+{
+	evt_ring_command(gsi, evt_ring_id, GSI_EVT_RESET);
+
+	evt_ring_command(gsi, evt_ring_id, GSI_EVT_DE_ALLOC);
+}
+
+/* Allocate an available event ring id */
+static int gsi_evt_ring_id_alloc(struct gsi *gsi)
+{
+	u32 evt_ring_id;
+
+	if (gsi->event_bitmap == ~0U)
+		return -ENOSPC;
+
+	evt_ring_id = ffz(gsi->event_bitmap);
+	gsi->event_bitmap |= BIT(evt_ring_id);
+
+	return (int)evt_ring_id;
+}
+
+/* Free a previously-allocated event ring id */
+static void gsi_evt_ring_id_free(struct gsi *gsi, u32 evt_ring_id)
+{
+	gsi->event_bitmap &= ~BIT(evt_ring_id);
+}
+
+/* Ring a channel doorbell, reporting the first un-filled entry */
+void gsi_channel_doorbell(struct gsi_channel *channel)
+{
+	struct gsi_ring *tre_ring = &channel->tre_ring;
+	u32 channel_id = gsi_channel_id(channel);
+	struct gsi *gsi = channel->gsi;
+	u32 val;
+
+	/* Note: index *must* be used modulo the ring count here */
+	val = gsi_ring_addr(tre_ring, tre_ring->index % tre_ring->count);
+	iowrite32(val, gsi->virt + GSI_CH_C_DOORBELL_0_OFFSET(channel_id));
+}
+
+/* Consult hardware, move any newly completed transactions to completed list */
+static void gsi_channel_update(struct gsi_channel *channel)
+{
+	u32 evt_ring_id = channel->evt_ring_id;
+	struct gsi *gsi = channel->gsi;
+	struct gsi_evt_ring *evt_ring;
+	struct gsi_trans *trans;
+	struct gsi_ring *ring;
+	u32 offset;
+	u32 index;
+
+	evt_ring = &gsi->evt_ring[evt_ring_id];
+	ring = &evt_ring->ring;
+
+	/* See if there's anything new to process; if not, we're done.  Note
+	 * that index always refers to an entry *within* the event ring.
+	 */
+	offset = GSI_EV_CH_E_CNTXT_4_OFFSET(evt_ring_id);
+	index = gsi_ring_index(ring, ioread32(gsi->virt + offset));
+	if (index == ring->index % ring->count)
+		return;
+
+	/* Get the transaction for the latest completed event.  Take a
+	 * reference to keep it from completing before we give the events
+	 * for this and previous transactions back to the hardware.
+	 */
+	trans = gsi_event_trans(channel, gsi_ring_virt(ring, index - 1));
+	refcount_inc(&trans->refcount);
+
+	/* For RX channels, update each completed transaction with the number
+	 * of bytes that were actually received.  For TX channels, report
+	 * the number of transactions and bytes this completion represents
+	 * up the network stack.
+	 */
+	if (channel->toward_ipa)
+		gsi_channel_tx_update(channel, trans);
+	else
+		gsi_evt_ring_rx_update(evt_ring, index);
+
+	gsi_trans_move_complete(trans);
+
+	/* Tell the hardware we've handled these events */
+	gsi_evt_ring_doorbell(channel->gsi, channel->evt_ring_id, index);
+
+	gsi_trans_free(trans);
+}
+
+/**
+ * gsi_channel_poll_one() - Return a single completed transaction on a channel
+ * @channel:	Channel to be polled
+ *
+ * @Return:	 Transaction pointer, or null if none are available
+ *
+ * This function returns the first entry on a channel's completed transaction
+ * list.  If that list is empty, the hardware is consulted to determine
+ * whether any new transactions have completed.  If so, they're moved to the
+ * completed list and the new first entry is returned.  If there are no more
+ * completed transactions, a null pointer is returned.
+ */
+static struct gsi_trans *gsi_channel_poll_one(struct gsi_channel *channel)
+{
+	struct gsi_trans *trans;
+
+	/* Get the first transaction from the completed list */
+	trans = gsi_channel_trans_complete(channel);
+	if (!trans) {
+		/* List is empty; see if there's more to do */
+		gsi_channel_update(channel);
+		trans = gsi_channel_trans_complete(channel);
+	}
+
+	if (trans)
+		gsi_trans_move_polled(trans);
+
+	return trans;
+}
+
+/**
+ * gsi_channel_poll() - NAPI poll function for a channel
+ * @napi:	NAPI structure for the channel
+ * @budget:	Budget supplied by NAPI core
+
+ * @Return:	 Number of items polled (<= budget)
+ *
+ * Single transactions completed by hardware are polled until either
+ * the budget is exhausted, or there are no more.  Each transaction
+ * polled is passed to gsi_trans_complete(), to perform remaining
+ * completion processing and retire/free the transaction.
+ */
+static int gsi_channel_poll(struct napi_struct *napi, int budget)
+{
+	struct gsi_channel *channel;
+	int count = 0;
+
+	channel = container_of(napi, struct gsi_channel, napi);
+	while (count < budget) {
+		struct gsi_trans *trans;
+
+		trans = gsi_channel_poll_one(channel);
+		if (!trans)
+			break;
+		gsi_trans_complete(trans);
+	}
+
+	if (count < budget) {
+		napi_complete(&channel->napi);
+		gsi_irq_event_enable(channel->gsi, channel->evt_ring_id);
+	}
+
+	return count;
+}
+
+/* The event bitmap represents which event ids are available for allocation.
+ * Set bits are not available, clear bits can be used.  This function
+ * initializes the map so all events supported by the hardware are available,
+ * then precludes any reserved events from being allocated.
+ */
+static u32 gsi_event_bitmap_init(u32 evt_ring_max)
+{
+	u32 event_bitmap = GENMASK(BITS_PER_LONG - 1, evt_ring_max);
+
+	return event_bitmap | GENMASK(GSI_MHI_ER_END, GSI_MHI_ER_START);
+}
+
+/* Setup function for event rings */
+static int gsi_evt_ring_setup(struct gsi *gsi)
+{
+	u32 evt_ring_max;
+	u32 evt_ring_id;
+
+	evt_ring_max = gsi_evt_ring_max(gsi);
+	dev_dbg(gsi->dev, "evt_ring_max %u\n", evt_ring_max);
+	if (evt_ring_max != GSI_EVT_RING_MAX)
+		return -EIO;
+
+	for (evt_ring_id = 0; evt_ring_id < GSI_EVT_RING_MAX; evt_ring_id++) {
+		struct gsi_evt_ring *evt_ring = &gsi->evt_ring[evt_ring_id];
+
+		evt_ring->state = gsi_evt_ring_state(gsi, evt_ring_id);
+		if (evt_ring->state != GSI_EVT_RING_STATE_NOT_ALLOCATED)
+			return -EIO;
+	}
+
+	return 0;
+}
+
+/* Inverse of gsi_evt_ring_setup() */
+static void gsi_evt_ring_teardown(struct gsi *gsi)
+{
+	/* Nothing to do */
+}
+
+/* Configure a channel's "scratch registers" for a particular protocol */
+static void gsi_channel_scratch_write(struct gsi_channel *channel)
+{
+	u32 channel_id = gsi_channel_id(channel);
+	union gsi_channel_scratch scr = { };
+	struct gsi_gpi_channel_scratch *gpi;
+	struct gsi *gsi = channel->gsi;
+	u32 val;
+
+	/* See comments above definition of gsi_gpi_channel_scratch */
+	gpi = &scr.gpi;
+	gpi->max_outstanding_tre = channel->data->tlv_count *
+					sizeof(struct gsi_tre);
+	gpi->outstanding_threshold = 2 * sizeof(struct gsi_tre);
+
+	val = scr.data.word1;
+	iowrite32(val, gsi->virt + GSI_CH_C_SCRATCH_0_OFFSET(channel_id));
+
+	val = scr.data.word2;
+	iowrite32(val, gsi->virt + GSI_CH_C_SCRATCH_1_OFFSET(channel_id));
+
+	val = scr.data.word3;
+	iowrite32(val, gsi->virt + GSI_CH_C_SCRATCH_2_OFFSET(channel_id));
+
+	/* We must preserve the upper 16 bits of the last scratch register.
+	 * The next sequence assumes those bits remain unchanged between the
+	 * read and the write.
+	 */
+	val = ioread32(gsi->virt + GSI_CH_C_SCRATCH_3_OFFSET(channel_id));
+	val = (scr.data.word4 & GENMASK(31, 16)) | (val & GENMASK(15, 0));
+	iowrite32(val, gsi->virt + GSI_CH_C_SCRATCH_3_OFFSET(channel_id));
+}
+
+/* Program a channel for use */
+static void gsi_channel_program(struct gsi_channel *channel, bool doorbell)
+{
+	size_t size = channel->tre_ring.count * sizeof(struct gsi_tre);
+	u32 channel_id = gsi_channel_id(channel);
+	struct gsi *gsi = channel->gsi;
+	u32 wrr_weight = 0;
+	u32 val = 0;
+
+	BUILD_BUG_ON(sizeof(struct gsi_tre) > field_max(ELEMENT_SIZE_FMASK));
+
+	val |= u32_encode_bits(GSI_CHANNEL_PROTOCOL_GPI, CHTYPE_PROTOCOL_FMASK);
+	if (channel->toward_ipa)
+		val |= CHTYPE_DIR_FMASK;
+	val |= u32_encode_bits(channel->evt_ring_id, ERINDEX_FMASK);
+	val |= u32_encode_bits(sizeof(struct gsi_tre), ELEMENT_SIZE_FMASK);
+	iowrite32(val, gsi->virt + GSI_CH_C_CNTXT_0_OFFSET(channel_id));
+
+	val = u32_encode_bits(size, R_LENGTH_FMASK);
+	iowrite32(val, gsi->virt + GSI_CH_C_CNTXT_1_OFFSET(channel_id));
+
+	/* The context 2 and 3 registers store the low-order and
+	 * high-order 32 bits of the address of the channel ring,
+	 * respectively.
+	 */
+	val = channel->tre_ring.addr & GENMASK(31, 0);
+	iowrite32(val, gsi->virt + GSI_CH_C_CNTXT_2_OFFSET(channel_id));
+
+	val = channel->tre_ring.addr >> 32;
+	iowrite32(val, gsi->virt + GSI_CH_C_CNTXT_3_OFFSET(channel_id));
+
+	if (channel->data->wrr_priority)
+		wrr_weight = field_max(WRR_WEIGHT_FMASK);
+	val = u32_encode_bits(wrr_weight, WRR_WEIGHT_FMASK);
+
+	/* Max prefetch is 1 segment (do not set MAX_PREFETCH_FMASK) */
+	if (doorbell)
+		val |= USE_DB_ENG_FMASK;
+	iowrite32(val, gsi->virt + GSI_CH_C_QOS_OFFSET(channel_id));
+}
+
+/* Configure a channel; we configure all channels to use GPI protocol */
+static void gsi_channel_config(struct gsi_channel *channel, bool db_enable)
+{
+	/* Start at the first TRE entry each time we configure the channel */
+	channel->tre_ring.index = 0;
+	gsi_channel_program(channel, db_enable);
+	gsi_channel_scratch_write(channel);
+}
+
+/* Setup function for a single channel */
+static int gsi_channel_setup_one(struct gsi_channel *channel)
+{
+	u32 evt_ring_id = channel->evt_ring_id;
+	struct gsi *gsi = channel->gsi;
+	u32 val;
+	int ret;
+
+	if (!gsi)
+		return 0;	/* Ignore uninitialized channels */
+
+	channel->state = gsi_channel_state(channel);
+	if (channel->state != GSI_CHANNEL_STATE_NOT_ALLOCATED)
+		return -EIO;
+
+	mutex_lock(&gsi->mutex);
+
+	ret = gsi_evt_ring_alloc_hw(gsi, evt_ring_id);
+	if (ret)
+		goto err_mutex_unlock;
+
+	gsi_channel_command(channel, GSI_CH_ALLOCATE);
+	gsi->channel_stats.allocate++;
+
+	ret = channel->state == GSI_CHANNEL_STATE_ALLOCATED ? 0 : -EIO;
+	if (ret)
+		goto err_free_evt_ring;
+
+	gsi_channel_config(channel, true);
+
+	mutex_unlock(&gsi->mutex);
+
+	if (channel->toward_ipa)
+		netif_tx_napi_add(&gsi->dummy_dev, &channel->napi,
+				  gsi_channel_poll, NAPI_POLL_WEIGHT);
+	else
+		netif_napi_add(&gsi->dummy_dev, &channel->napi,
+			       gsi_channel_poll, NAPI_POLL_WEIGHT);
+
+	/* Enable the event interrupt (clear it first in case pending) */
+	val = BIT(evt_ring_id);
+	iowrite32(val, gsi->virt + GSI_CNTXT_SRC_IEOB_IRQ_CLR_OFFSET);
+	gsi_irq_event_enable(gsi, evt_ring_id);
+
+	return 0;
+
+err_free_evt_ring:
+	gsi_evt_ring_free_hw(gsi, evt_ring_id);
+err_mutex_unlock:
+	mutex_unlock(&gsi->mutex);
+
+	return ret;
+}
+
+/* Inverse of gsi_channel_setup_one() */
+static void gsi_channel_teardown_one(struct gsi_channel *channel)
+{
+	u32 evt_ring_id = channel->evt_ring_id;
+	struct gsi *gsi = channel->gsi;
+
+	if (!gsi)
+		return;
+
+	gsi_irq_event_disable(gsi, evt_ring_id);
+
+	netif_napi_del(&channel->napi);
+
+	mutex_lock(&gsi->mutex);
+
+	gsi_channel_command(channel, GSI_CH_DE_ALLOC);
+	gsi->channel_stats.free++;
+
+	gsi_evt_ring_free_hw(gsi, evt_ring_id);
+
+	mutex_unlock(&gsi->mutex);
+
+	gsi_channel_trans_exit(channel);
+}
+
+/* Setup function for channels */
+static int gsi_channel_setup(struct gsi *gsi)
+{
+	u32 channel_max;
+	u32 channel_id;
+	int ret;
+
+	channel_max = gsi_channel_max(gsi);
+	dev_dbg(gsi->dev, "channel_max %u\n", channel_max);
+	if (channel_max != GSI_CHANNEL_MAX)
+		return -EIO;
+
+	ret = gsi_evt_ring_setup(gsi);
+	if (ret)
+		return ret;
+
+	gsi_irq_enable(gsi);
+
+	for (channel_id = 0; channel_id < GSI_CHANNEL_MAX; channel_id++) {
+		ret = gsi_channel_setup_one(&gsi->channel[channel_id]);
+		if (ret)
+			goto err_unwind;
+	}
+
+	return 0;
+
+err_unwind:
+	while (channel_id--)
+		gsi_channel_teardown_one(&gsi->channel[channel_id]);
+	gsi_irq_enable(gsi);
+	gsi_evt_ring_teardown(gsi);
+
+	return ret;
+}
+
+/* Inverse of gsi_channel_setup() */
+static void gsi_channel_teardown(struct gsi *gsi)
+{
+	u32 channel_id;
+
+	for (channel_id = 0; channel_id < GSI_CHANNEL_MAX; channel_id++) {
+		struct gsi_channel *channel = &gsi->channel[channel_id];
+
+		gsi_channel_teardown_one(channel);
+	}
+	gsi_irq_disable(gsi);
+	gsi_evt_ring_teardown(gsi);
+}
+
+/* Setup function for GSI.  GSI firmware must be loaded and initialized */
+int gsi_setup(struct gsi *gsi)
+{
+	u32 val;
+
+	/* Here is where we first touch the GSI hardware */
+	val = ioread32(gsi->virt + GSI_GSI_STATUS_OFFSET);
+	if (!(val & ENABLED_FMASK)) {
+		dev_err(gsi->dev, "GSI has not been enabled\n");
+		return -EIO;
+	}
+
+	/* Initialize the error log */
+	iowrite32(0, gsi->virt + GSI_ERROR_LOG_OFFSET);
+
+	/* Writing 1 indicates IRQ interrupts; 0 would be MSI */
+	iowrite32(1, gsi->virt + GSI_CNTXT_INTSET_OFFSET);
+
+	return gsi_channel_setup(gsi);
+}
+
+/* Inverse of gsi_setup() */
+void gsi_teardown(struct gsi *gsi)
+{
+	gsi_channel_teardown(gsi);
+}
+
+/* Initialize a channel's event ring */
+static int gsi_channel_evt_ring_init(struct gsi_channel *channel)
+{
+	struct gsi *gsi = channel->gsi;
+	struct gsi_evt_ring *evt_ring;
+	int ret;
+
+	ret = gsi_evt_ring_id_alloc(gsi);
+	if (ret < 0)
+		return ret;
+	channel->evt_ring_id = ret;
+
+	evt_ring = &gsi->evt_ring[channel->evt_ring_id];
+	evt_ring->channel = channel;
+
+	ret = gsi_ring_alloc(gsi, &evt_ring->ring, channel->data->event_count);
+	if (ret)
+		goto err_free_evt_ring_id;
+
+	return 0;
+
+err_free_evt_ring_id:
+	gsi_evt_ring_id_free(gsi, channel->evt_ring_id);
+
+	return ret;
+}
+
+/* Inverse of gsi_channel_evt_ring_init() */
+static void gsi_channel_evt_ring_exit(struct gsi_channel *channel)
+{
+	struct gsi *gsi = channel->gsi;
+	struct gsi_evt_ring *evt_ring;
+
+	evt_ring = &gsi->evt_ring[channel->evt_ring_id];
+	gsi_ring_free(gsi, &evt_ring->ring);
+	gsi_evt_ring_id_free(gsi, channel->evt_ring_id);
+}
+
+/* Init function for event rings */
+static void gsi_evt_ring_init(struct gsi *gsi)
+{
+	u32 evt_ring_id;
+
+	BUILD_BUG_ON(GSI_EVT_RING_MAX >= BITS_PER_LONG);
+
+	gsi->event_bitmap = gsi_event_bitmap_init(GSI_EVT_RING_MAX);
+	gsi->event_enable_bitmap = 0;
+	for (evt_ring_id = 0; evt_ring_id < GSI_EVT_RING_MAX; evt_ring_id++)
+		init_completion(&gsi->evt_ring[evt_ring_id].completion);
+}
+
+/* Inverse of gsi_evt_ring_init() */
+static void gsi_evt_ring_exit(struct gsi *gsi)
+{
+	/* Nothing to do */
+}
+
+/* Init function for a single channel */
+static int
+gsi_channel_init_one(struct gsi *gsi, const struct gsi_ipa_endpoint_data *data)
+{
+	struct gsi_channel *channel;
+	int ret;
+
+	if (data->ee_id != GSI_EE_AP)
+		return 0;	/* Ignore non-AP channels */
+
+	if (data->channel_id >= GSI_CHANNEL_MAX) {
+		dev_err(gsi->dev, "bad channel id %u (must be less than %u)\n",
+			data->channel_id, GSI_CHANNEL_MAX);
+		return -EINVAL;
+	}
+
+	/* The value 256 here is arbitrary, and much higher than expected */
+	if (!data->channel.tlv_count || data->channel.tlv_count > 256) {
+		dev_err(gsi->dev, "bad tlv_count %u (must be 1..256)\n",
+			data->channel.tlv_count);
+		return -EINVAL;
+	}
+
+	if (!is_power_of_2(data->channel.tre_count)) {
+		dev_err(gsi->dev, "bad tre_count %u (must be power of 2)\n",
+			data->channel.tre_count);
+		return -EINVAL;
+	}
+
+	if (!is_power_of_2(data->channel.event_count)) {
+		dev_err(gsi->dev, "bad event_count %u (must be power of 2)\n",
+			data->channel.event_count);
+		return -EINVAL;
+	}
+
+	channel = &gsi->channel[data->channel_id];
+	memset(channel, 0, sizeof(*channel));
+
+	channel->gsi = gsi;
+	channel->toward_ipa = data->toward_ipa;
+	channel->data = &data->channel;
+
+	init_completion(&channel->completion);
+
+	ret = gsi_channel_evt_ring_init(channel);
+	if (ret)
+		return ret;
+
+	ret = gsi_ring_alloc(gsi, &channel->tre_ring, channel->data->tre_count);
+	if (ret)
+		goto err_channel_evt_ring_exit;
+
+	ret = gsi_channel_trans_init(channel);
+	if (ret)
+		goto err_ring_free;
+
+	return 0;
+
+err_ring_free:
+	gsi_ring_free(gsi, &channel->tre_ring);
+err_channel_evt_ring_exit:
+	gsi_channel_evt_ring_exit(channel);
+
+	return ret;
+}
+
+/* Inverse of gsi_channel_init_one() */
+static void gsi_channel_exit_one(struct gsi_channel *channel)
+{
+	gsi_channel_trans_exit(channel);
+	gsi_ring_free(channel->gsi, &channel->tre_ring);
+	gsi_channel_evt_ring_exit(channel);
+}
+
+/* Init function for channels */
+static int gsi_channel_init(struct gsi *gsi, u32 data_count,
+			    const struct gsi_ipa_endpoint_data *data)
+{
+	int ret = 0;
+	u32 i;
+
+	gsi_evt_ring_init(gsi);
+	for (i = 0; i < data_count; i++) {
+		ret = gsi_channel_init_one(gsi, &data[i]);
+		if (ret)
+			break;
+	}
+
+	return ret;
+}
+
+/* Inverse of gsi_channel_init() */
+static void gsi_channel_exit(struct gsi *gsi)
+{
+	u32 channel_id;
+
+	for (channel_id = 0; channel_id < GSI_CHANNEL_MAX; channel_id++)
+		gsi_channel_exit_one(&gsi->channel[channel_id]);
+	gsi_evt_ring_exit(gsi);
+}
+
+/* Init function for GSI.  GSI hardware does not need to be "ready" */
+int gsi_init(struct gsi *gsi, struct platform_device *pdev, u32 data_count,
+	     const struct gsi_ipa_endpoint_data *data)
+{
+	struct resource *res;
+	resource_size_t size;
+	unsigned int irq;
+	int ret;
+
+	gsi->dev = &pdev->dev;
+
+	/* The GSI layer performs NAPI on all endpoints.  NAPI requires a
+	 * network device structure, but the GSI layer does not have one,
+	 * so we must create a dummy network device for this purpose.
+	 */
+	init_dummy_netdev(&gsi->dummy_dev);
+
+	/* Get GSI memory range and map it */
+	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "gsi");
+	if (!res)
+		return -ENXIO;
+
+	size = resource_size(res);
+	if (res->start > U32_MAX || size > U32_MAX - res->start)
+		return -EINVAL;
+
+	gsi->virt = ioremap(res->start, size);
+	if (!gsi->virt)
+		return -ENOMEM;
+
+	mutex_init(&gsi->mutex);
+
+	ret = platform_get_irq_byname(pdev, "gsi");
+	if (ret < 0)
+		goto err_unmap_virt;
+	irq = ret;
+
+	ret = request_irq(irq, gsi_isr, 0, "gsi", gsi);
+	if (ret)
+		goto err_unmap_virt;
+	gsi->irq = irq;
+
+	ret = enable_irq_wake(gsi->irq);
+	if (ret)
+		dev_err(gsi->dev, "error %d enabling gsi wake irq\n", ret);
+	gsi->irq_wake_enabled = ret ? 0 : 1;
+
+	ret = gsi_channel_init(gsi, data_count, data);
+	if (ret)
+		goto err_mutex_destroy;
+
+	return 0;
+
+err_mutex_destroy:
+	if (gsi->irq_wake_enabled)
+		(void)disable_irq_wake(gsi->irq);
+	free_irq(gsi->irq, gsi);
+	mutex_destroy(&gsi->mutex);
+err_unmap_virt:
+	iounmap(gsi->virt);
+
+	return ret;
+}
+
+/* Inverse of gsi_init() */
+void gsi_exit(struct gsi *gsi)
+{
+	gsi_channel_exit(gsi);
+
+	if (gsi->irq_wake_enabled)
+		(void)disable_irq_wake(gsi->irq);
+	free_irq(gsi->irq, gsi);
+	mutex_destroy(&gsi->mutex);
+	iounmap(gsi->virt);
+}
+
+/* Returns the maximum number of pending transactions on a channel */
+u32 gsi_channel_trans_max(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+
+	return channel->data->tre_count;
+}
+
+/* Returns the maximum number of TREs in a single transaction for a channel */
+u32 gsi_channel_trans_tre_max(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+
+	return channel->data->tlv_count;
+}
+
+/* Wait for all transaction activity on a channel to complete */
+void gsi_channel_trans_quiesce(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_trans *trans;
+
+	/* Get the last transaction, and wait for it to complete */
+	trans = gsi_channel_trans_last(gsi, channel_id);
+	if (trans) {
+		wait_for_completion(&trans->completion);
+		gsi_trans_free(trans);
+	}
+}
+
+/* Make a channel operational */
+int gsi_channel_start(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+
+	if (channel->state != GSI_CHANNEL_STATE_ALLOCATED &&
+	    channel->state != GSI_CHANNEL_STATE_STOP_IN_PROC &&
+	    channel->state != GSI_CHANNEL_STATE_STOPPED) {
+		dev_err(gsi->dev, "channel %u bad state %u\n", channel_id,
+			(u32)channel->state);
+		return -ENOTSUPP;
+	}
+
+	napi_enable(&channel->napi);
+
+	mutex_lock(&gsi->mutex);
+
+	gsi_channel_command(channel, GSI_CH_START);
+	gsi->channel_stats.start++;
+
+	mutex_unlock(&gsi->mutex);
+
+	return 0;
+}
+
+/* Stop an operational channel */
+int gsi_channel_stop(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+	int ret;
+
+	if (channel->state == GSI_CHANNEL_STATE_STOPPED)
+		return 0;
+
+	if (channel->state != GSI_CHANNEL_STATE_STARTED &&
+	    channel->state != GSI_CHANNEL_STATE_STOP_IN_PROC &&
+	    channel->state != GSI_CHANNEL_STATE_ERROR) {
+		dev_err(gsi->dev, "channel %u bad state %u\n", channel_id,
+			(u32)channel->state);
+		return -ENOTSUPP;
+	}
+
+	gsi_channel_trans_quiesce(gsi, channel_id);
+
+	mutex_lock(&gsi->mutex);
+
+	gsi_channel_command(channel, GSI_CH_STOP);
+	gsi->channel_stats.stop++;
+
+	mutex_unlock(&gsi->mutex);
+
+	if (channel->state == GSI_CHANNEL_STATE_STOPPED)
+		ret = 0;
+	else if (channel->state == GSI_CHANNEL_STATE_STOP_IN_PROC)
+		ret = -EAGAIN;
+	else
+		ret = -EIO;
+
+	if (!ret)
+		napi_disable(&channel->napi);
+
+	return ret;
+}
+
+/* Reset and reconfigure a GSI channel (possibly leaving doorbell disabled) */
+int gsi_channel_reset(struct gsi *gsi, u32 channel_id, bool db_enable)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+
+	if (channel->state != GSI_CHANNEL_STATE_STOPPED) {
+		dev_err(gsi->dev, "channel %u bad state %u\n", channel_id,
+			(u32)channel->state);
+		return -ENOTSUPP;
+	}
+
+	/* In case the reset follows stop, need to wait 1 msec */
+	usleep_range(USEC_PER_MSEC, 2 * USEC_PER_MSEC);
+
+	mutex_lock(&gsi->mutex);
+
+	gsi_channel_command(channel, GSI_CH_RESET);
+	gsi->channel_stats.reset++;
+
+	/* workaround: reset RX channels again */
+	if (!channel->toward_ipa) {
+		usleep_range(USEC_PER_MSEC, 2 * USEC_PER_MSEC);
+		gsi_channel_command(channel, GSI_CH_RESET);
+	}
+
+	gsi_channel_config(channel, db_enable);
+
+	/* Cancel pending transactions before the channel is started again */
+	gsi_channel_trans_cancel_pending(channel);
+
+	mutex_unlock(&gsi->mutex);
+
+	return 0;
+}
Signature · 1 line
2.20.1
author
#13 lore raw

PATCH v2 8/17 soc: qcom: ipa: GSI transactions

In reply to Alex Elder

This patch implements GSI transactions. A GSI transaction is a structure that represents a single request (consisting of one or more TREs) sent to the GSI hardware. The last TRE in a transaction includes a flag requesting that the GSI interrupt the AP to notify that it has completed.

TREs are executed and completed strictly in order. For this reason, the completion of a single TRE implies that all previous TREs (in particular all of those "earlier" in a transaction) have completed.

Whenever there is a need to send a request (a set of TREs) to the IPA, a GSI transaction is allocated, specifying the number of TREs that will be required. Details of the request (e.g. transfer offsets and length) are represented by in a Linux scatterlist array that is incorporated in the transaction structure.

Once "filled," the transaction is committed. The GSI transaction layer performs all needed mapping (and unmapping) for DMA, and issues the request to the hardware. When the hardware signals that the request has completed, a callback function allows for cleanup or followup activity to be performed before the transaction is freed.

---
 drivers/net/ipa/gsi_trans.c | 624 ++++++++++++++++++++++++++++++++++++
 drivers/net/ipa/gsi_trans.h | 116 +++++++
 2 files changed, 740 insertions(+)
 create mode 100644 drivers/net/ipa/gsi_trans.c
 create mode 100644 drivers/net/ipa/gsi_trans.h
drivers/net/ipa/gsi_trans.c new file +624 -0
new file mode 100644
index 000000000000..267e33093554
@@ -0,0 +1,624 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/bits.h>
+#include <linux/bitfield.h>
+#include <linux/refcount.h>
+#include <linux/scatterlist.h>
+
+#include "gsi.h"
+#include "gsi_private.h"
+#include "gsi_trans.h"
+#include "ipa_gsi.h"
+#include "ipa_data.h"
+#include "ipa_cmd.h"
+
+/**
+ * DOC: GSI Transactions
+ *
+ * A GSI transaction abstracts the behavior of a GSI channel by representing
+ * everything about a related group of data transfers in a single structure.
+ * Most details of interaction with the GSI hardware are managed by the GSI
+ * transaction core, allowing users to simply describe transfers to be
+ * performed.  When a transaction has completed a callback function
+ * (dependent on the type of endpoint associated with the channel) allows
+ * cleanup of resources associated with the transaction.
+ *
+ * To perform a data transfer (or a related set of them), a user of the GSI
+ * transaction interface allocates a transaction, indicating the number of
+ * TREs required (one per data transfer).  If sufficient TREs are available,
+ * they are reserved for use in the transaction and the allocation succeeds.
+ * This way exhaustion of the available TREs in a channel ring is detected
+ * as early as possible.  All resources required to complete a transaction
+ * are allocated at transaction allocation time.
+ *
+ * Transfers performed as part of a transaction are represented in an array
+ * of Linux scatterlist structures.  This array is allocated with the
+ * transaction, and its entries must be initialized using standard
+ * scatterlist functions (such as sg_init_one() or skb_to_sgvec()).
+ *
+ * Once a transaction's scatterlist structures have been initialized, the
+ * transaction is committed.  The GSI transaction layer is responsible for
+ * DMA mapping (and unmapping) memory described in the transaction's
+ * scatterlist array.  The only way committing a transaction fails is if
+ * this DMA mapping step returns an error.  Otherwise, ownership of the
+ * entire transaction is transferred to the GSI transaction core.  The GSI
+ * transaction code formats the content of the scatterlist array into the
+ * channel ring buffer and informs the hardware that new TREs are available
+ * to process.
+ *
+ * The last TRE in each transaction is marked to interrupt the AP when the
+ * GSI hardware has completed it.  Because transfers described by TREs are
+ * performed strictly in order, signaling the completion of just the last
+ * TRE in the transaction is sufficient to indicate the full transaction
+ * is complete.
+ *
+ * When a transaction is complete, ipa_gsi_trans_complete() is called by the
+ * GSI code into the IPA layer, allowing it to perform any final cleanup
+ * required before the transaction is freed.
+ */
+
+/* gsi_tre->flags mask values (in CPU byte order) */
+#define GSI_TRE_FLAGS_CHAIN_FMASK	GENMASK(0, 0)
+#define GSI_TRE_FLAGS_IEOB_FMASK	GENMASK(8, 8)
+#define GSI_TRE_FLAGS_IEOT_FMASK	GENMASK(9, 9)
+#define GSI_TRE_FLAGS_BEI_FMASK		GENMASK(10, 10)
+#define GSI_TRE_FLAGS_TYPE_FMASK	GENMASK(23, 16)
+
+/* Hardware values representing a transfer element type */
+enum gsi_tre_type {
+	GSI_RE_XFER	= 0x2,
+	GSI_RE_IMMD_CMD	= 0x3,
+	GSI_RE_NOP	= 0x4,
+};
+
+/* Map a given ring entry index to the transaction associated with it */
+static void gsi_channel_trans_map(struct gsi_channel *channel, u32 index,
+				  struct gsi_trans *trans)
+{
+	/* Note: index *must* be used modulo the ring count here */
+	channel->trans_info.map[index % channel->tre_ring.count] = trans;
+}
+
+/* Return the transaction mapped to a given ring entry */
+struct gsi_trans *
+gsi_channel_trans_mapped(struct gsi_channel *channel, u32 index)
+{
+	/* Note: index *must* be used modulo the ring count here */
+	return channel->trans_info.map[index % channel->tre_ring.count];
+}
+
+/* Return the oldest completed transaction for a channel (or null) */
+struct gsi_trans *gsi_channel_trans_complete(struct gsi_channel *channel)
+{
+	return list_first_entry_or_null(&channel->trans_info.complete,
+					struct gsi_trans, links);
+}
+
+/* Move a transaction from the allocated list to the pending list */
+static void gsi_trans_move_pending(struct gsi_trans *trans)
+{
+	struct gsi_channel *channel = &trans->gsi->channel[trans->channel_id];
+	struct gsi_trans_info *trans_info = &channel->trans_info;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	list_move_tail(&trans->links, &trans_info->pending);
+
+	spin_unlock_bh(&trans_info->spinlock);
+}
+
+/* Move a transaction and all of its predecessors from the pending list
+ * to the completed list.
+ */
+void gsi_trans_move_complete(struct gsi_trans *trans)
+{
+	struct gsi_channel *channel = &trans->gsi->channel[trans->channel_id];
+	struct gsi_trans_info *trans_info = &channel->trans_info;
+	struct list_head list;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	/* Move this transaction and all predecessors to completed list */
+	list_cut_position(&list, &trans_info->pending, &trans->links);
+	list_splice_tail(&list, &trans_info->complete);
+
+	spin_unlock_bh(&trans_info->spinlock);
+}
+
+/* Move a transaction from the completed list to the polled list */
+void gsi_trans_move_polled(struct gsi_trans *trans)
+{
+	struct gsi_channel *channel = &trans->gsi->channel[trans->channel_id];
+	struct gsi_trans_info *trans_info = &channel->trans_info;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	list_move_tail(&trans->links, &trans_info->polled);
+
+	spin_unlock_bh(&trans_info->spinlock);
+}
+
+/* Return the last (most recent) transaction allocated on a channel */
+struct gsi_trans *gsi_channel_trans_last(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_trans_info *trans_info;
+	struct gsi_trans *trans;
+	struct list_head *list;
+
+	trans_info = &gsi->channel[channel_id].trans_info;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	/* Find the last list to which a transaction was added */
+	if (!list_empty(&trans_info->alloc))
+		list = &trans_info->alloc;
+	else if (!list_empty(&trans_info->pending))
+		list = &trans_info->pending;
+	else if (!list_empty(&trans_info->complete))
+		list = &trans_info->complete;
+	else if (!list_empty(&trans_info->polled))
+		list = &trans_info->polled;
+	else
+		list = NULL;
+
+	if (list) {
+		/* The last entry on this list is the last one allocated.
+		 * Grab a reference so it can be waited for.
+		 */
+		trans = list_last_entry(list, struct gsi_trans, links);
+		refcount_inc(&trans->refcount);
+	} else {
+		trans = NULL;
+	}
+
+	spin_unlock_bh(&trans_info->spinlock);
+
+	return trans;
+}
+
+/* Reserve some number of TREs on a channel.  Returns true if successful */
+static bool
+gsi_trans_tre_reserve(struct gsi_trans_info *trans_info, u32 tre_count)
+{
+	int avail = atomic_read(&trans_info->tre_avail);
+	int new;
+
+	do {
+		new = avail - (int)tre_count;
+		if (unlikely(new < 0))
+			return false;
+	} while (!atomic_try_cmpxchg(&trans_info->tre_avail, &avail, new));
+
+	return true;
+}
+
+/* Release previously-reserved TRE entries to a channel */
+static void
+gsi_trans_tre_release(struct gsi_trans_info *trans_info, u32 tre_count)
+{
+	atomic_add(tre_count, &trans_info->tre_avail);
+}
+
+/* Allocate a GSI transaction on a channel */
+struct gsi_trans *
+gsi_channel_trans_alloc(struct gsi *gsi, u32 channel_id, u32 tre_count)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+	struct gsi_trans_info *trans_info;
+	struct gsi_trans *trans;
+	u32 which;
+
+	/* Caller should know the limit is gsi_channel_trans_max() */
+	if (WARN_ON(tre_count > channel->data->tlv_count))
+		return NULL;
+
+	trans_info = &channel->trans_info;
+
+	/* We reserve the TREs now, but consume them at commit time.
+	 * If there aren't enough available, we're done.
+	 */
+	if (!gsi_trans_tre_reserve(trans_info, tre_count))
+		return NULL;
+
+	/* Allocate the transaction and initialize it */
+	which = trans_info->pool_free++ % trans_info->pool_count;
+	trans = &trans_info->pool[which];
+
+	trans->gsi = gsi;
+	trans->channel_id = channel_id;
+	refcount_set(&trans->refcount, 1);
+	trans->tre_count = tre_count;
+	init_completion(&trans->completion);
+
+	/* We're reusing, so make sure all fields are reinitialized */
+	trans->dev = gsi->dev;
+	trans->result = 0;	/* Success assumed unless overwritten */
+	trans->data = NULL;
+
+	/* Allocate the scatter/gather entries it will use.  If what's
+	 * needed would cross the end-of-pool boundary, allocate them
+	 * from the beginning of the pool.
+	 */
+	if (tre_count > trans_info->sg_pool_count - trans_info->sg_pool_free)
+		trans_info->sg_pool_free = 0;
+	trans->sgl = &trans_info->sg_pool[trans_info->sg_pool_free];
+	trans->sgc = tre_count;
+	trans_info->sg_pool_free += tre_count;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	list_add_tail(&trans->links, &trans_info->alloc);
+
+	spin_unlock_bh(&trans_info->spinlock);
+
+	return trans;
+}
+
+/* Free a previously-allocated transaction (used only in case of error) */
+void gsi_trans_free(struct gsi_trans *trans)
+{
+	struct gsi_trans_info *trans_info;
+
+	if (!refcount_dec_and_test(&trans->refcount))
+		return;
+
+	trans_info = &trans->gsi->channel[trans->channel_id].trans_info;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	list_del(&trans->links);
+
+	spin_unlock_bh(&trans_info->spinlock);
+
+	gsi_trans_tre_release(trans_info, trans->tre_count);
+}
+
+/* Compute the length/opcode value to use for a TRE */
+static __le16 gsi_tre_len_opcode(enum ipa_cmd_opcode opcode, u32 len)
+{
+	return opcode == IPA_CMD_NONE ? cpu_to_le16((u16)len)
+				      : cpu_to_le16((u16)opcode);
+}
+
+/* Compute the flags value to use for a given TRE */
+static __le32 gsi_tre_flags(bool last_tre, bool bei, enum ipa_cmd_opcode opcode)
+{
+	enum gsi_tre_type tre_type;
+	u32 tre_flags;
+
+	tre_type = opcode == IPA_CMD_NONE ? GSI_RE_XFER : GSI_RE_IMMD_CMD;
+	tre_flags = u32_encode_bits(tre_type, GSI_TRE_FLAGS_TYPE_FMASK);
+
+	/* Last TRE contains interrupt flags */
+	if (last_tre) {
+		/* All transactions end in a transfer completion interrupt */
+		tre_flags |= GSI_TRE_FLAGS_IEOT_FMASK;
+		/* Don't interrupt when outbound commands are acknowledged */
+		if (bei)
+			tre_flags |= GSI_TRE_FLAGS_BEI_FMASK;
+	} else {	/* All others indicate there's more to come */
+		tre_flags |= GSI_TRE_FLAGS_CHAIN_FMASK;
+	}
+
+	return cpu_to_le32(tre_flags);
+}
+
+static void gsi_trans_tre_fill(struct gsi_tre *dest_tre, dma_addr_t addr,
+			       u32 len, bool last_tre, bool bei,
+			       enum ipa_cmd_opcode opcode)
+{
+	struct gsi_tre tre;
+
+	tre.addr = cpu_to_le64(addr);
+	tre.len_opcode = gsi_tre_len_opcode(opcode, len);
+	tre.reserved = 0;
+	tre.flags = gsi_tre_flags(last_tre, bei, opcode);
+
+	/* ARM64 can write 16 bytes as a unit with a single instruction.
+	 * Doing the assignment this way is an attempt to make that happen.
+	 */
+	*dest_tre = tre;
+}
+
+/* Issue a command to read a single byte from a channel */
+int gsi_trans_read_byte(struct gsi *gsi, u32 channel_id, dma_addr_t addr)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+	struct gsi_trans_info *trans_info;
+	struct gsi_ring *tre_ring;
+	struct gsi_tre *dest_tre;
+	struct gsi_ring *ring;
+
+	trans_info = &channel->trans_info;
+
+	/* First reserve the TRE, if possible */
+	if (!gsi_trans_tre_reserve(trans_info, 1))
+		return -EBUSY;
+
+	/* Now allocate the next TRE, fill it, and tell the hardware */
+	tre_ring = &channel->tre_ring;
+	ring = &gsi->evt_ring[channel->evt_ring_id].ring;
+
+	dest_tre = gsi_ring_virt(tre_ring, tre_ring->index);
+	gsi_trans_tre_fill(dest_tre, addr, 1, true, false, IPA_CMD_NONE);
+
+	tre_ring->index++;
+	gsi_channel_doorbell(channel);
+
+	return 0;
+}
+
+/* Mark a gsi_trans_read_byte() request done */
+void gsi_trans_read_byte_done(struct gsi *gsi, u32 channel_id)
+{
+	struct gsi_channel *channel = &gsi->channel[channel_id];
+
+	gsi_trans_tre_release(&channel->trans_info, 1);
+}
+
+/**
+ * __gsi_trans_commit() - Common GSI transaction commit code
+ * @trans:	Transaction to commit
+ * @opcode:	Immediate command opcode, or IPA_CMD_NONE
+ * @ring_db:	Whether to tell the hardware about these queued transfers
+ *
+ * @Return:	0 if successful, or a negative error code
+ *
+ * Maps the transactions's scatterlist array for DMA, and returns -ENOMEM
+ * if that fails.  Formats channel ring TRE entries based on the content of
+ * the scatterlist.  Maps a transaction pointer to the last ring entry used
+ * for the transaction, so it can be recovered when it completes.  Moves
+ * the transaction to the pending list.  Finally, updates the channel ring
+ * pointer and optionally rings the doorbell.
+ */
+static int __gsi_trans_commit(struct gsi_trans *trans,
+			      enum ipa_cmd_opcode opcode, bool ring_db)
+{
+	struct gsi_channel *channel = &trans->gsi->channel[trans->channel_id];
+	struct gsi_ring *tre_ring = &channel->tre_ring;
+	enum dma_data_direction direction;
+	bool bei = channel->toward_ipa;
+	struct gsi_tre *dest_tre;
+	struct scatterlist *sg;
+	struct gsi_ring *ring;
+	u32 byte_count = 0;
+	u32 avail;
+	int ret;
+	u32 i;
+
+	direction = channel->toward_ipa ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+	ret = dma_map_sg(trans->dev, trans->sgl, trans->sgc, direction);
+	if (!ret)
+		return -ENOMEM;
+
+	ring = &channel->gsi->evt_ring[channel->evt_ring_id].ring;
+
+	/* Consume the entries.  If we cross the end of the ring while
+	 * filling them we'll switch to the beginning to finish.
+	 */
+	avail = ring->count - tre_ring->index % tre_ring->count;
+	dest_tre = gsi_ring_virt(tre_ring, tre_ring->index);
+	for_each_sg(trans->sgl, sg, trans->sgc, i) {
+		bool last_tre = i == trans->tre_count - 1;
+		dma_addr_t addr = sg_dma_address(sg);
+		u32 len = sg_dma_len(sg);
+
+		byte_count += len;
+		if (!avail--)
+			dest_tre = gsi_ring_virt(tre_ring, 0);
+
+		gsi_trans_tre_fill(dest_tre, addr, len, last_tre, bei, opcode);
+		dest_tre++;
+	}
+	tre_ring->index += trans->tre_count;
+
+	if (channel->toward_ipa) {
+		/* We record TX bytes when they are sent */
+		trans->len = byte_count;
+		trans->trans_count = channel->trans_count;
+		trans->byte_count = channel->byte_count;
+		channel->trans_count++;
+		channel->byte_count += byte_count;
+	}
+
+	/* Associate the last TRE with the transaction */
+	gsi_channel_trans_map(channel, tre_ring->index - 1, trans);
+
+	gsi_trans_move_pending(trans);
+
+	/* Ring doorbell if requested, or if all TREs are allocated */
+	if (ring_db || !atomic_read(&channel->trans_info.tre_avail)) {
+		/* Report what we're handing off to hardware for TX channels */
+		if (channel->toward_ipa)
+			gsi_channel_tx_queued(channel);
+		gsi_channel_doorbell(channel);
+	}
+
+	return 0;
+}
+
+/* Commit a GSI transaction */
+int gsi_trans_commit(struct gsi_trans *trans, bool ring_db)
+{
+	return __gsi_trans_commit(trans, IPA_CMD_NONE, ring_db);
+}
+
+/* Commit a GSI command transaction and wait for it to complete */
+int gsi_trans_commit_command(struct gsi_trans *trans,
+			     enum ipa_cmd_opcode opcode)
+{
+	int ret;
+
+	refcount_inc(&trans->refcount);
+
+	ret = __gsi_trans_commit(trans, opcode, true);
+	if (ret)
+		goto out_free_trans;
+
+	wait_for_completion(&trans->completion);
+
+out_free_trans:
+	gsi_trans_free(trans);
+
+	return ret;
+}
+
+/* Commit a GSI command transaction, wait for it to complete, with timeout */
+int gsi_trans_commit_command_timeout(struct gsi_trans *trans,
+				     enum ipa_cmd_opcode opcode,
+				     unsigned long timeout)
+{
+	unsigned long timeout_jiffies = msecs_to_jiffies(timeout);
+	unsigned long remaining;
+	int ret;
+
+	refcount_inc(&trans->refcount);
+
+	ret = __gsi_trans_commit(trans, opcode, true);
+	if (ret)
+		goto out_free_trans;
+
+	remaining = wait_for_completion_timeout(&trans->completion,
+						timeout_jiffies);
+out_free_trans:
+	gsi_trans_free(trans);
+
+	return ret ? ret : remaining ? 0 : -ETIMEDOUT;
+}
+
+/* Return a channel's next completed transaction (or NULL) */
+void gsi_trans_complete(struct gsi_trans *trans)
+{
+	struct gsi_channel *channel = &trans->gsi->channel[trans->channel_id];
+	enum dma_data_direction direction;
+
+	direction = channel->toward_ipa ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+
+	dma_unmap_sg(trans->dev, trans->sgl, trans->sgc, direction);
+
+	ipa_gsi_trans_complete(trans);
+
+	complete(&trans->completion);
+
+	gsi_trans_free(trans);
+}
+
+/* Cancel a channel's pending transactions */
+void gsi_channel_trans_cancel_pending(struct gsi_channel *channel)
+{
+	struct gsi_trans_info *trans_info = &channel->trans_info;
+	u32 evt_ring_id = channel->evt_ring_id;
+	struct gsi *gsi = channel->gsi;
+	struct gsi_trans *trans;
+	struct gsi_ring *ring;
+
+	ring = &gsi->evt_ring[evt_ring_id].ring;
+
+	spin_lock_bh(&trans_info->spinlock);
+
+	list_for_each_entry(trans, &trans_info->pending, links)
+		trans->result = -ECANCELED;
+
+	list_splice_tail_init(&trans_info->pending, &trans_info->complete);
+
+	spin_unlock_bh(&trans_info->spinlock);
+
+	/* Schedule NAPI polling to complete the cancelled transactions */
+	napi_schedule(&channel->napi);
+}
+
+/* Initialize a channel's GSI transaction info */
+int gsi_channel_trans_init(struct gsi_channel *channel)
+{
+	struct gsi_trans_info *trans_info = &channel->trans_info;
+	u32 tre_count = channel->data->tre_count;
+
+	trans_info->map = kcalloc(tre_count, sizeof(*trans_info->map),
+				  GFP_KERNEL);
+	if (!trans_info->map)
+		return -ENOMEM;
+
+	/* We will never need more transactions than there are TRE
+	 * entries in the transfer ring.  For that reason, we can
+	 * preallocate an array of (at least) that many transactions,
+	 * and use a single free index to determine the next one
+	 * available for allocation.
+	 */
+	trans_info->pool_count = tre_count;
+	trans_info->pool = kcalloc(trans_info->pool_count,
+				   sizeof(*trans_info->pool), GFP_KERNEL);
+	if (!trans_info->pool)
+		goto err_free_map;
+	/* If we get extra memory from the allocator, use it */
+	trans_info->pool_count =
+		ksize(trans_info->pool) / sizeof(*trans_info->pool);
+	trans_info->pool_free = 0;
+
+	/* While transactions are allocated one at a time, a transaction
+	 * can have multiple TREs.  The number of TRE entries in a single
+	 * transaction is limited by the number of TLV FIFO entries the
+	 * channel has.  We reserve TREs when a transaction is allocated,
+	 * but we don't actually use/fill them until the transaction is
+	 * committed.
+	 *
+	 * A transaction uses a scatterlist array to represent the data
+	 * transfers implemented by the transaction.  Each scatterlist
+	 * element is used to fill a single TRE when the transaction is
+	 * committed.  As a result, we need the same number of scatterlist
+	 * elements as there are TREs in the transfer ring, and we can
+	 * preallocate them in a pool.
+	 *
+	 * If we allocate a few (tlv_count - 1) extra entries in our pool,
+	 * we can always satisfy requests without ever worrying about
+	 * straddling the end of the array.  If there aren't enough
+	 * entries starting at the free index, we just allocate free
+	 * entries from the beginning of the pool.
+	 */
+	trans_info->sg_pool_count = tre_count + channel->data->tlv_count - 1;
+	trans_info->sg_pool = kcalloc(trans_info->sg_pool_count,
+				      sizeof(*trans_info->sg_pool), GFP_KERNEL);
+	if (!trans_info->sg_pool)
+		goto err_free_pool;
+	/* Use any extra memory we get from the allocator */
+	trans_info->sg_pool_count =
+		ksize(trans_info->sg_pool) / sizeof(*trans_info->sg_pool);
+	trans_info->sg_pool_free = 0;
+
+	/* The tre_avail field limits the number of outstanding transactions.
+	 * In theory we should be able use all of the TREs in the ring.  But
+	 * in practice, doing that caused the hardware to report running out
+	 * of event ring slots for writing completion information.  So give
+	 * the poor hardware a break, and allow one less than the maximum.
+	 */
+	atomic_set(&trans_info->tre_avail, tre_count - 1);
+
+	spin_lock_init(&trans_info->spinlock);
+	INIT_LIST_HEAD(&trans_info->alloc);
+	INIT_LIST_HEAD(&trans_info->pending);
+	INIT_LIST_HEAD(&trans_info->complete);
+	INIT_LIST_HEAD(&trans_info->polled);
+
+	return 0;
+
+err_free_pool:
+	kfree(trans_info->pool);
+err_free_map:
+	kfree(trans_info->map);
+
+	return -ENOMEM;
+}
+
+/* Inverse of gsi_channel_trans_init() */
+void gsi_channel_trans_exit(struct gsi_channel *channel)
+{
+	struct gsi_trans_info *trans_info = &channel->trans_info;
+
+	kfree(trans_info->sg_pool);
+	kfree(trans_info->pool);
+	kfree(trans_info->map);
+}
drivers/net/ipa/gsi_trans.h new file +116 -0

116 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#14 lore raw

PATCH v2 9/17 soc: qcom: ipa: IPA interface to GSI

In reply to Alex Elder

This patch provides interface functions supplied by the IPA layer that are called from the GSI layer. One function is called when a GSI transaction has completed. The others allow the GSI layer to inform the IPA layer when the hardware has been told it has new TREs to execute, and when the hardware has indicated transactions have completed.

---
 drivers/net/ipa/ipa_gsi.c | 48 ++++++++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_gsi.h | 49 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_gsi.c
 create mode 100644 drivers/net/ipa/ipa_gsi.h
drivers/net/ipa/ipa_gsi.c new file +48 -0
new file mode 100644
index 000000000000..7f8d74688c1e
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+
+#include "gsi_trans.h"
+#include "ipa.h"
+#include "ipa_endpoint.h"
+
+void ipa_gsi_trans_complete(struct gsi_trans *trans)
+{
+	struct ipa *ipa = container_of(trans->gsi, struct ipa, gsi);
+	struct ipa_endpoint *endpoint;
+
+	endpoint = ipa->endpoint_map[trans->channel_id];
+	if (endpoint == ipa->command_endpoint)
+		return;		/* Nothing to do for commands */
+
+	if (endpoint->toward_ipa)
+		ipa_endpoint_skb_tx_complete(trans);
+	else
+		ipa_endpoint_rx_complete(trans);
+}
+
+void ipa_gsi_channel_tx_queued(struct gsi *gsi, u32 channel_id, u32 count,
+			       u32 byte_count)
+{
+	struct ipa *ipa = container_of(gsi, struct ipa, gsi);
+	struct ipa_endpoint *endpoint;
+
+	endpoint = ipa->endpoint_map[channel_id];
+	if (endpoint->netdev)
+		netdev_sent_queue(endpoint->netdev, byte_count);
+}
+
+void ipa_gsi_channel_tx_completed(struct gsi *gsi, u32 channel_id, u32 count,
+				  u32 byte_count)
+{
+	struct ipa *ipa = container_of(gsi, struct ipa, gsi);
+	struct ipa_endpoint *endpoint;
+
+	endpoint = ipa->endpoint_map[channel_id];
+	if (endpoint->netdev)
+		netdev_completed_queue(endpoint->netdev, count, byte_count);
+}
drivers/net/ipa/ipa_gsi.h new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#15 lore raw

PATCH v2 10/17 soc: qcom: ipa: IPA endpoints

In reply to Alex Elder

This patch includes the code implementing an IPA endpoint. This is the primary abstraction implemented by the IPA. An endpoint is one end of a network connection between two entities physically connected to the IPA. Specifically, the AP and the modem implement endpoints, and an (AP endpoint, modem endpoint) pair implements the transfer of network data in one direction between the AP and modem.

Endpoints are built on top of GSI channels, but IPA endpoints represent the higher-level functionality that the IPA provides. Data can be sent through a GSI channel, but it is the IPA endpoint that represents what is on the "other end" to receive that data. Other functionality, including aggregation, checksum offload and (at some future date) IP routing and filtering are all associated with the IPA endpoint.

---
 drivers/net/ipa/ipa_endpoint.c | 1283 ++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_endpoint.h |   97 +++
 2 files changed, 1380 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_endpoint.c
 create mode 100644 drivers/net/ipa/ipa_endpoint.h
drivers/net/ipa/ipa_endpoint.c new file +1283 -0
new file mode 100644
index 000000000000..0185db35033d
@@ -0,0 +1,1283 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+#include <linux/bitfield.h>
+#include <linux/if_rmnet.h>
+
+#include "gsi.h"
+#include "gsi_trans.h"
+#include "ipa.h"
+#include "ipa_data.h"
+#include "ipa_endpoint.h"
+#include "ipa_cmd.h"
+#include "ipa_mem.h"
+#include "ipa_netdev.h"
+
+#define atomic_dec_not_zero(v)	atomic_add_unless((v), -1, 0)
+
+#define IPA_REPLENISH_BATCH	16
+
+#define IPA_RX_BUFFER_SIZE	(PAGE_SIZE << IPA_RX_BUFFER_ORDER)
+#define IPA_RX_BUFFER_ORDER	1	/* 8KB endpoint RX buffers (2 pages) */
+
+/* The amount of RX buffer space consumed by standard skb overhead */
+#define IPA_RX_BUFFER_OVERHEAD	(PAGE_SIZE - SKB_MAX_ORDER(NET_SKB_PAD, 0))
+
+#define IPA_ENDPOINT_STOP_RETRY_MAX		10
+#define IPA_ENDPOINT_STOP_RX_SIZE		1	/* bytes */
+
+#define IPA_ENDPOINT_RESET_AGGR_RETRY_MAX	3
+#define IPA_AGGR_TIME_LIMIT_DEFAULT		1	/* milliseconds */
+
+/** enum ipa_status_opcode - status element opcode hardware values */
+enum ipa_status_opcode {
+	IPA_STATUS_OPCODE_PACKET		= 0x01,
+	IPA_STATUS_OPCODE_NEW_FRAG_RULE		= 0x02,
+	IPA_STATUS_OPCODE_DROPPED_PACKET	= 0x04,
+	IPA_STATUS_OPCODE_SUSPENDED_PACKET	= 0x08,
+	IPA_STATUS_OPCODE_LOG			= 0x10,
+	IPA_STATUS_OPCODE_DCMP			= 0x20,
+	IPA_STATUS_OPCODE_PACKET_2ND_PASS	= 0x40,
+};
+
+/** enum ipa_status_exception - status element exception type */
+enum ipa_status_exception {
+	IPA_STATUS_EXCEPTION_NONE,
+	IPA_STATUS_EXCEPTION_DEAGGR,
+	IPA_STATUS_EXCEPTION_IPTYPE,
+	IPA_STATUS_EXCEPTION_PACKET_LENGTH,
+	IPA_STATUS_EXCEPTION_PACKET_THRESHOLD,
+	IPA_STATUS_EXCEPTION_FRAG_RULE_MISS,
+	IPA_STATUS_EXCEPTION_SW_FILT,
+	IPA_STATUS_EXCEPTION_NAT,
+	IPA_STATUS_EXCEPTION_IPV6CT,
+	IPA_STATUS_EXCEPTION_MAX,
+};
+
+/**
+ * struct ipa_status - Abstracted IPA status element
+ * @opcode:		Status element type
+ * @exception:		The first exception that took place
+ * @pkt_len:		Payload length
+ * @dst_endpoint:	Destination endpoint
+ * @metadata:		32-bit metadata value used by packet
+ * @rt_miss:		Flag; if 1, indicates there was a routing rule miss
+ *
+ * Note that the hardware status element supplies additional information
+ * that is currently unused.
+ */
+struct ipa_status {
+	enum ipa_status_opcode opcode;
+	enum ipa_status_exception exception;
+	u32 pkt_len;
+	u32 dst_endpoint;
+	u32 metadata;
+	u32 rt_miss;
+};
+
+/* Field masks for struct ipa_status_raw structure fields */
+
+#define IPA_STATUS_SRC_IDX_FMASK		GENMASK(4, 0)
+
+#define IPA_STATUS_DST_IDX_FMASK		GENMASK(4, 0)
+
+#define IPA_STATUS_FLAGS1_FLT_LOCAL_FMASK	GENMASK(0, 0)
+#define IPA_STATUS_FLAGS1_FLT_HASH_FMASK	GENMASK(1, 1)
+#define IPA_STATUS_FLAGS1_FLT_GLOBAL_FMASK	GENMASK(2, 2)
+#define IPA_STATUS_FLAGS1_FLT_RET_HDR_FMASK	GENMASK(3, 3)
+#define IPA_STATUS_FLAGS1_FLT_RULE_ID_FMASK	GENMASK(13, 4)
+#define IPA_STATUS_FLAGS1_RT_LOCAL_FMASK	GENMASK(14, 14)
+#define IPA_STATUS_FLAGS1_RT_HASH_FMASK		GENMASK(15, 15)
+#define IPA_STATUS_FLAGS1_UCP_FMASK		GENMASK(16, 16)
+#define IPA_STATUS_FLAGS1_RT_TBL_IDX_FMASK	GENMASK(21, 17)
+#define IPA_STATUS_FLAGS1_RT_RULE_ID_FMASK	GENMASK(31, 22)
+
+#define IPA_STATUS_FLAGS2_NAT_HIT_FMASK		GENMASK_ULL(0, 0)
+#define IPA_STATUS_FLAGS2_NAT_ENTRY_IDX_FMASK	GENMASK_ULL(13, 1)
+#define IPA_STATUS_FLAGS2_NAT_TYPE_FMASK	GENMASK_ULL(15, 14)
+#define IPA_STATUS_FLAGS2_TAG_INFO_FMASK	GENMASK_ULL(63, 16)
+
+#define IPA_STATUS_FLAGS3_SEQ_NUM_FMASK		GENMASK(7, 0)
+#define IPA_STATUS_FLAGS3_TOD_CTR_FMASK		GENMASK(31, 8)
+
+#define IPA_STATUS_FLAGS4_HDR_LOCAL_FMASK	GENMASK(0, 0)
+#define IPA_STATUS_FLAGS4_HDR_OFFSET_FMASK	GENMASK(10, 1)
+#define IPA_STATUS_FLAGS4_FRAG_HIT_FMASK	GENMASK(11, 11)
+#define IPA_STATUS_FLAGS4_FRAG_RULE_FMASK	GENMASK(15, 12)
+#define IPA_STATUS_FLAGS4_HW_SPECIFIC_FMASK	GENMASK(31, 16)
+
+/* Status element provided by hardware */
+struct ipa_status_raw {
+	u8 opcode;
+	u8 exception;
+	u16 mask;
+	u16 pkt_len;
+	u8 endp_src_idx;	/* Only bottom 5 bits valid */
+	u8 endp_dst_idx;	/* Only bottom 5 bits valid */
+	u32 metadata;
+	u32 flags1;
+	u64 flags2;
+	u32 flags3;
+	u32 flags4;
+};
+
+static void ipa_endpoint_replenish(struct ipa_endpoint *endpoint, bool add_one);
+
+/* suspend_delay represents suspend for RX, delay for TX endpoints */
+bool ipa_endpoint_init_ctrl(struct ipa_endpoint *endpoint, bool suspend_delay)
+{
+	u32 offset = IPA_REG_ENDP_INIT_CTRL_N_OFFSET(endpoint->endpoint_id);
+	u32 mask;
+	u32 val;
+
+	mask = endpoint->toward_ipa ? ENDP_DELAY_FMASK : ENDP_SUSPEND_FMASK;
+
+	val = ioread32(endpoint->ipa->reg_virt + offset);
+	if (suspend_delay == !!(val & mask))
+		return false;	/* Already set to desired state */
+
+	val ^= mask;
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+
+	return true;
+}
+
+static void ipa_endpoint_init_cfg(struct ipa_endpoint *endpoint)
+{
+	u32 offset = IPA_REG_ENDP_INIT_CFG_N_OFFSET(endpoint->endpoint_id);
+	u32 val = 0;
+
+	/* FRAG_OFFLOAD_EN is 0 */
+	if (endpoint->data->config.checksum) {
+		if (endpoint->toward_ipa) {
+			u32 checksum_offset;
+
+			val |= u32_encode_bits(IPA_CS_OFFLOAD_UL,
+					       CS_OFFLOAD_EN_FMASK);
+			/* Checksum header offset is in 4-byte units */
+			checksum_offset = sizeof(struct rmnet_map_header);
+			checksum_offset /= sizeof(u32);
+			val |= u32_encode_bits(checksum_offset,
+					       CS_METADATA_HDR_OFFSET_FMASK);
+		} else {
+			val |= u32_encode_bits(IPA_CS_OFFLOAD_DL,
+					       CS_OFFLOAD_EN_FMASK);
+		}
+	} else {
+		val |= u32_encode_bits(IPA_CS_OFFLOAD_NONE,
+				       CS_OFFLOAD_EN_FMASK);
+	}
+	/* CS_GEN_QMB_MASTER_SEL is 0 */
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+static void ipa_endpoint_init_hdr(struct ipa_endpoint *endpoint)
+{
+	u32 offset = IPA_REG_ENDP_INIT_HDR_N_OFFSET(endpoint->endpoint_id);
+	u32 val = 0;
+
+	if (endpoint->data->config.qmap) {
+		size_t header_size = sizeof(struct rmnet_map_header);
+
+		if (endpoint->toward_ipa && endpoint->data->config.checksum)
+			header_size += sizeof(struct rmnet_map_ul_csum_header);
+
+		val |= u32_encode_bits(header_size, HDR_LEN_FMASK);
+		/* metadata is the 4 byte rmnet_map header itself */
+		val |= HDR_OFST_METADATA_VALID_FMASK;
+		val |= u32_encode_bits(0, HDR_OFST_METADATA_FMASK);
+		/* HDR_ADDITIONAL_CONST_LEN is 0; (IPA->AP only) */
+		if (!endpoint->toward_ipa) {
+			u32 size_offset = offsetof(struct rmnet_map_header,
+						   pkt_len);
+
+			val |= HDR_OFST_PKT_SIZE_VALID_FMASK;
+			val |= u32_encode_bits(size_offset,
+					       HDR_OFST_PKT_SIZE_FMASK);
+		}
+		/* HDR_A5_MUX is 0 */
+		/* HDR_LEN_INC_DEAGG_HDR is 0 */
+		/* HDR_METADATA_REG_VALID is 0; (AP->IPA only) */
+	}
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+static void ipa_endpoint_init_hdr_ext(struct ipa_endpoint *endpoint)
+{
+	u32 offset = IPA_REG_ENDP_INIT_HDR_EXT_N_OFFSET(endpoint->endpoint_id);
+	u32 pad_align = endpoint->data->config.rx.pad_align;
+	u32 val = 0;
+
+	val |= HDR_ENDIANNESS_FMASK;		/* big endian */
+	val |= HDR_TOTAL_LEN_OR_PAD_VALID_FMASK;
+	/* HDR_TOTAL_LEN_OR_PAD is 0 (pad, not total_len) */
+	/* HDR_PAYLOAD_LEN_INC_PADDING is 0 */
+	/* HDR_TOTAL_LEN_OR_PAD_OFFSET is 0 */
+	if (!endpoint->toward_ipa)
+		val |= u32_encode_bits(pad_align, HDR_PAD_TO_ALIGNMENT_FMASK);
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+/**
+ * Generate a metadata mask value that will select only the mux_id
+ * field in an rmnet_map header structure.  The mux_id is at offset
+ * 1 byte from the beginning of the structure, but the metadata
+ * value is treated as a 4-byte unit.  So this mask must be computed
+ * with endianness in mind.  Note that ipa_endpoint_init_hdr_metadata_mask()
+ * will convert this value to the proper byte order.
+ *
+ * Marked __always_inline because this is really computing a
+ * constant value.
+ */
+static __always_inline __be32 ipa_rmnet_mux_id_metadata_mask(void)
+{
+	size_t mux_id_offset = offsetof(struct rmnet_map_header, mux_id);
+	u32 mux_id_mask = 0;
+	u8 *bytes;
+
+	bytes = (u8 *)&mux_id_mask;
+	bytes[mux_id_offset] = 0xff;	/* mux_id is 1 byte */
+
+	return cpu_to_be32(mux_id_mask);
+}
+
+static void ipa_endpoint_init_hdr_metadata_mask(struct ipa_endpoint *endpoint)
+{
+	u32 endpoint_id = endpoint->endpoint_id;
+	u32 val = 0;
+	u32 offset;
+
+	offset = IPA_REG_ENDP_INIT_HDR_METADATA_MASK_N_OFFSET(endpoint_id);
+
+	if (!endpoint->toward_ipa && endpoint->data->config.qmap)
+		val = ipa_rmnet_mux_id_metadata_mask();
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+/* Compute the aggregation size value to use for a given buffer size */
+static u32 ipa_aggr_size_kb(u32 rx_buffer_size)
+{
+	BUILD_BUG_ON(IPA_RX_BUFFER_SIZE >
+		     field_max(AGGR_BYTE_LIMIT_FMASK) * SZ_1K +
+		     IPA_MTU + IPA_RX_BUFFER_OVERHEAD);
+
+	/* Because we don't have the "hard byte limit" enabled, we
+	 * need to make sure there's enough space in the buffer to
+	 * receive a complete MTU (plus normal skb overhead) beyond
+	 * the aggregated size limit we specify.
+	 */
+	rx_buffer_size -= IPA_MTU + IPA_RX_BUFFER_OVERHEAD;
+
+	return rx_buffer_size / SZ_1K;
+}
+
+static void ipa_endpoint_init_aggr(struct ipa_endpoint *endpoint)
+{
+	const struct ipa_endpoint_config_data *config = &endpoint->data->config;
+	u32 offset = IPA_REG_ENDP_INIT_AGGR_N_OFFSET(endpoint->endpoint_id);
+	u32 val = 0;
+
+	if (config->aggregation) {
+		if (!endpoint->toward_ipa) {
+			u32 aggr_size = ipa_aggr_size_kb(IPA_RX_BUFFER_SIZE);
+
+			val |= u32_encode_bits(IPA_ENABLE_AGGR, AGGR_EN_FMASK);
+			val |= u32_encode_bits(IPA_GENERIC, AGGR_TYPE_FMASK);
+			val |= u32_encode_bits(aggr_size,
+					       AGGR_BYTE_LIMIT_FMASK);
+			val |= u32_encode_bits(IPA_AGGR_TIME_LIMIT_DEFAULT,
+					       AGGR_TIME_LIMIT_FMASK);
+			val |= u32_encode_bits(0, AGGR_PKT_LIMIT_FMASK);
+			if (config->rx.aggr_close_eof)
+				val |= AGGR_SW_EOF_ACTIVE_FMASK;
+			/* AGGR_HARD_BYTE_LIMIT_ENABLE is 0 */
+		} else {
+			val |= u32_encode_bits(IPA_ENABLE_DEAGGR,
+					       AGGR_EN_FMASK);
+			val |= u32_encode_bits(IPA_QCMAP, AGGR_TYPE_FMASK);
+			/* other fields ignored */
+		}
+		/* AGGR_FORCE_CLOSE is 0 */
+	} else {
+		val |= u32_encode_bits(IPA_BYPASS_AGGR, AGGR_EN_FMASK);
+		/* other fields ignored */
+	}
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+static void ipa_endpoint_init_mode(struct ipa_endpoint *endpoint)
+{
+	u32 offset = IPA_REG_ENDP_INIT_MODE_N_OFFSET(endpoint->endpoint_id);
+	u32 val = 0;
+
+	if (endpoint->toward_ipa && endpoint->data->config.dma_mode) {
+		u32 dma_endpoint_id = endpoint->data->config.dma_endpoint;
+
+		val |= u32_encode_bits(IPA_DMA, MODE_FMASK);
+		val |= u32_encode_bits(dma_endpoint_id, DEST_PIPE_INDEX_FMASK);
+	} else {
+		val |= u32_encode_bits(IPA_BASIC, MODE_FMASK);
+	}
+	/* Other bitfields unspecified (and 0) */
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+static void ipa_endpoint_init_deaggr(struct ipa_endpoint *endpoint)
+{
+	u32 offset = IPA_REG_ENDP_INIT_DEAGGR_N_OFFSET(endpoint->endpoint_id);
+	u32 val = 0;
+
+	/* DEAGGR_HDR_LEN is 0 */
+	/* PACKET_OFFSET_VALID is 0 */
+	/* PACKET_OFFSET_LOCATION is ignored (not valid) */
+	/* MAX_PACKET_LEN is 0 (not enforced) */
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+static void ipa_endpoint_init_seq(struct ipa_endpoint *endpoint)
+{
+	u32 offset = IPA_REG_ENDP_INIT_SEQ_N_OFFSET(endpoint->endpoint_id);
+	u32 seq_type = endpoint->data->seq_type;
+	u32 val = 0;
+
+	val |= u32_encode_bits(seq_type & 0xf, HPS_SEQ_TYPE_FMASK);
+	val |= u32_encode_bits((seq_type >> 4) & 0xf, DPS_SEQ_TYPE_FMASK);
+	/* HPS_REP_SEQ_TYPE is 0 */
+	/* DPS_REP_SEQ_TYPE is 0 */
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+/* Complete transaction initiated in ipa_endpoint_skb_tx() */
+void ipa_endpoint_skb_tx_complete(struct gsi_trans *trans)
+{
+	struct sk_buff *skb = trans->data;
+
+	dev_kfree_skb_any(skb);
+}
+
+/**
+ * ipa_endpoint_skb_tx() - Transmit a socket buffer
+ * @endpoint:	Endpoint pointer
+ * @skb:	Socket buffer to send
+ *
+ * Returns:	0 if successful, or a negative error code
+ */
+int ipa_endpoint_skb_tx(struct ipa_endpoint *endpoint, struct sk_buff *skb)
+{
+	struct gsi_trans *trans;
+	u32 nr_frags;
+	int ret;
+
+	/* Make sure source endpoint's TLV FIFO has enough entries to
+	 * hold the linear portion of the skb and all its fragments.
+	 * If not, see if we can linearize it before giving up.
+	 */
+	nr_frags = skb_shinfo(skb)->nr_frags;
+	if (1 + nr_frags > endpoint->trans_tre_max) {
+		if (skb_linearize(skb))
+			return -ENOMEM;
+		nr_frags = 0;
+	}
+
+	trans = gsi_channel_trans_alloc(&endpoint->ipa->gsi,
+					endpoint->channel_id, nr_frags + 1);
+	if (!trans)
+		return -EBUSY;
+	trans->data = skb;
+
+	ret = skb_to_sgvec(skb, trans->sgl, 0, skb->len);
+	if (ret < 0)
+		goto err_trans_free;
+	trans->sgc = ret;
+
+	ret = gsi_trans_commit(trans, !netdev_xmit_more());
+	if (ret)
+		goto err_trans_free;
+	return 0;
+
+err_trans_free:
+	gsi_trans_free(trans);
+
+	return -ENOMEM;
+}
+
+static void ipa_endpoint_status(struct ipa_endpoint *endpoint)
+{
+	const struct ipa_endpoint_config_data *config = &endpoint->data->config;
+	enum ipa_endpoint_id endpoint_id = endpoint->endpoint_id;
+	u32 val = 0;
+	u32 offset;
+
+	offset = IPA_REG_ENDP_STATUS_N_OFFSET(endpoint_id);
+
+	if (endpoint->data->config.status_enable) {
+		val |= STATUS_EN_FMASK;
+		if (endpoint->toward_ipa) {
+			u32 status_endpoint_id = config->tx.status_endpoint;
+
+			val |= u32_encode_bits(status_endpoint_id,
+					       STATUS_ENDP_FMASK);
+		}
+		/* STATUS_LOCATION is 0 (status element precedes packet) */
+		/* STATUS_PKT_SUPPRESS_FMASK */
+	}
+
+	iowrite32(val, endpoint->ipa->reg_virt + offset);
+}
+
+static void ipa_endpoint_skb_copy(struct ipa_endpoint *endpoint,
+				  void *data, u32 len, u32 extra)
+{
+	struct sk_buff *skb;
+
+	skb = __dev_alloc_skb(len, GFP_ATOMIC);
+	if (skb) {
+		skb_put(skb, len);
+		memcpy(skb->data, data, len);
+		skb->truesize += extra;
+	}
+
+	/* Now receive it, or drop it if there's no netdev */
+	if (endpoint->netdev)
+		ipa_netdev_skb_rx(endpoint->netdev, skb);
+	else if (skb)
+		dev_kfree_skb_any(skb);
+}
+
+static void ipa_endpoint_skb_build(struct ipa_endpoint *endpoint,
+				   struct page *page, u32 len)
+{
+	struct sk_buff *skb;
+
+	/* assert(len <= SKB_WITH_OVERHEAD(IPA_RX_BUFFER_SIZE-NET_SKB_PAD)); */
+	skb = build_skb(page_address(page), IPA_RX_BUFFER_SIZE);
+	if (skb) {
+		/* Reserve the headroom and account for the data */
+		skb_reserve(skb, NET_SKB_PAD);
+		skb_put(skb, len);
+	}
+
+	/* Now receive it, or drop it if there's no netdev */
+	if (endpoint->netdev)
+		ipa_netdev_skb_rx(endpoint->netdev, skb);
+	else if (skb)
+		dev_kfree_skb_any(skb);
+
+	/* If no socket buffer took the pages, free them */
+	if (!skb)
+		__free_pages(page, IPA_RX_BUFFER_ORDER);
+}
+
+/* Maps an exception type returned in a ipa_status_raw structure
+ * to the ipa_status_exception value that represents it in
+ * the exception field of a ipa_status structure.  Returns
+ * IPA_STATUS_EXCEPTION_MAX for an unrecognized value.
+ */
+static enum ipa_status_exception exception_map(u8 exception, bool is_ipv6)
+{
+	switch (exception) {
+	case 0x00:	return IPA_STATUS_EXCEPTION_NONE;
+	case 0x01:	return IPA_STATUS_EXCEPTION_DEAGGR;
+	case 0x04:	return IPA_STATUS_EXCEPTION_IPTYPE;
+	case 0x08:	return IPA_STATUS_EXCEPTION_PACKET_LENGTH;
+	case 0x10:	return IPA_STATUS_EXCEPTION_FRAG_RULE_MISS;
+	case 0x20:	return IPA_STATUS_EXCEPTION_SW_FILT;
+	case 0x40:	return is_ipv6 ? IPA_STATUS_EXCEPTION_IPV6CT
+				       : IPA_STATUS_EXCEPTION_NAT;
+	default:	return IPA_STATUS_EXCEPTION_MAX;
+	}
+}
+
+/* A rule miss is indicated as an all-1's value in the rt_rule_id
+ * or flt_rule_id field of the ipa_status structure.
+ */
+static bool ipa_rule_miss_id(u32 id)
+{
+	return id == field_max(IPA_STATUS_FLAGS1_RT_RULE_ID_FMASK);
+}
+
+size_t ipa_status_parse(struct ipa_status *status, void *data, u32 count)
+{
+	const struct ipa_status_raw *status_raw = data;
+	bool is_ipv6;
+	u32 val;
+
+	BUILD_BUG_ON(sizeof(*status_raw) % 4);
+	if (WARN_ON(count < sizeof(*status_raw)))
+		return 0;
+
+	status->opcode = status_raw->opcode;
+	is_ipv6 = status_raw->mask & BIT(7) ? false : true;
+	status->exception = exception_map(status_raw->exception, is_ipv6);
+	status->pkt_len = status_raw->pkt_len;
+	val = u32_get_bits(status_raw->endp_dst_idx, IPA_STATUS_DST_IDX_FMASK);
+	status->dst_endpoint = val;
+	status->metadata = status_raw->metadata;
+	val = u32_get_bits(status_raw->flags1,
+			   IPA_STATUS_FLAGS1_RT_RULE_ID_FMASK);
+	status->rt_miss = ipa_rule_miss_id(val) ? 1 : 0;
+
+	return sizeof(*status_raw);
+}
+
+/* The format of a packet status element is the same for several status
+ * types (opcodes).  The NEW_FRAG_RULE, LOG, DCMP (decompression) types
+ * aren't currently supported
+ */
+static bool ipa_status_format_packet(enum ipa_status_opcode opcode)
+{
+	switch (opcode) {
+	case IPA_STATUS_OPCODE_PACKET:
+	case IPA_STATUS_OPCODE_DROPPED_PACKET:
+	case IPA_STATUS_OPCODE_SUSPENDED_PACKET:
+	case IPA_STATUS_OPCODE_PACKET_2ND_PASS:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static bool ipa_endpoint_status_skip(struct ipa_endpoint *endpoint,
+				     struct ipa_status *status)
+{
+	if (!ipa_status_format_packet(status->opcode))
+		return true;
+	if (!status->pkt_len)
+		return true;
+	if (status->dst_endpoint != endpoint->endpoint_id)
+		return true;
+
+	return false;	/* Don't skip this packet, process it */
+}
+
+static void ipa_endpoint_status_parse(struct ipa_endpoint *endpoint,
+				      struct page *page, u32 total_len)
+{
+	void *data = page_address(page) + NET_SKB_PAD;
+	u32 unused = IPA_RX_BUFFER_SIZE - total_len;
+	u32 resid = total_len;
+
+	while (resid) {
+		struct ipa_status status;
+		bool drop_packet = false;
+		size_t status_size;
+		u32 align;
+		u32 len;
+
+		status_size = ipa_status_parse(&status, data, resid);
+
+		/* Skip over status packets that lack packet data */
+		if (ipa_endpoint_status_skip(endpoint, &status)) {
+			data += status_size;
+			resid -= status_size;
+			continue;
+		}
+
+		/* Packet data follows the status structure.  Unless
+		 * the packet failed to match a routing rule, or it
+		 * had a deaggregation exception, we'll consume it.
+		 */
+		if (status.exception == IPA_STATUS_EXCEPTION_NONE) {
+			if (status.rt_miss)
+				drop_packet = true;
+		} else if (status.exception == IPA_STATUS_EXCEPTION_DEAGGR) {
+			drop_packet = true;
+		}
+
+		/* Compute the amount of buffer space consumed by the
+		 * packet, including the status element.  If the hardware
+		 * is configured to pad packet data to an aligned boundary,
+		 * account for that.  And if checksum offload is is enabled
+		 * a trailer containing computed checksum information will
+		 * be appended.
+		 */
+		align = endpoint->data->config.rx.pad_align ? : 1;
+		len = status_size + ALIGN(status.pkt_len, align);
+		if (endpoint->data->config.checksum)
+			len += sizeof(struct rmnet_map_dl_csum_trailer);
+
+		/* Charge the new packet with a proportional fraction of
+		 * the unused space in the original receive buffer.
+		 * XXX Charge a proportion of the *whole* receive buffer?
+		 */
+		if (!drop_packet) {
+			u32 extra = unused * len / total_len;
+			void *data2 = data + status_size;
+			u32 len2 = status.pkt_len;
+
+			/* Client receives only packet data (no status) */
+			ipa_endpoint_skb_copy(endpoint, data2, len2, extra);
+		}
+
+		/* Consume status and the full packet it describes */
+		data += len;
+		resid -= len;
+	}
+
+	__free_pages(page, IPA_RX_BUFFER_ORDER);
+}
+
+/* Complete transaction initiated in ipa_endpoint_replenish_one() */
+void ipa_endpoint_rx_complete(struct gsi_trans *trans)
+{
+	struct page *page = trans->data;
+	struct ipa_endpoint *endpoint;
+	struct ipa *ipa;
+
+	ipa = container_of(trans->gsi, struct ipa, gsi);
+	endpoint = ipa->endpoint_map[trans->channel_id];
+
+	ipa_endpoint_replenish(endpoint, true);
+
+	if (trans->result == -ECANCELED) {
+		__free_pages(page, IPA_RX_BUFFER_ORDER);
+		return;
+	}
+
+	/* Parse or build a socket buffer using the actual received length */
+	if (endpoint->data->config.status_enable)
+		ipa_endpoint_status_parse(endpoint, page, trans->len);
+	else
+		ipa_endpoint_skb_build(endpoint, page, trans->len);
+}
+
+static int ipa_endpoint_replenish_one(struct ipa_endpoint *endpoint)
+{
+	struct gsi_trans *trans;
+	bool doorbell = false;
+	struct page *page;
+	u32 offset;
+	u32 len;
+
+	page = dev_alloc_pages(IPA_RX_BUFFER_ORDER);
+	if (!page)
+		return -ENOMEM;
+	offset = NET_SKB_PAD;
+	len = IPA_RX_BUFFER_SIZE - offset;
+
+	trans = gsi_channel_trans_alloc(&endpoint->ipa->gsi,
+					endpoint->channel_id, 1);
+	if (!trans)
+		goto err_page_free;
+	trans->data = page;
+
+	/* Set up and map a scatterlist entry representing the buffer */
+	sg_init_table(trans->sgl, trans->sgc);
+	sg_set_page(trans->sgl, page, len, offset);
+
+	if (++endpoint->replenish_ready == IPA_REPLENISH_BATCH) {
+		doorbell = true;
+		endpoint->replenish_ready = 0;
+	}
+
+	if (!gsi_trans_commit(trans, doorbell))
+		return 0;
+
+err_page_free:
+	__free_pages(page, IPA_RX_BUFFER_ORDER);
+
+	return -ENOMEM;
+}
+
+/**
+ * ipa_endpoint_replenish() - Replenish the Rx packets cache.
+ *
+ * Allocate RX packet wrapper structures with maximal socket buffers
+ * for an endpoint.  These are supplied to the hardware, which fills
+ * them with incoming data.
+ */
+static void ipa_endpoint_replenish(struct ipa_endpoint *endpoint, bool add_one)
+{
+	struct gsi *gsi;
+	u32 backlog;
+
+	if (add_one) {
+		if (endpoint->replenish_enabled)
+			atomic_inc(&endpoint->replenish_backlog);
+		else
+			atomic_inc(&endpoint->replenish_saved);
+	}
+
+	if (!endpoint->replenish_enabled)
+		return;
+
+	while (atomic_dec_not_zero(&endpoint->replenish_backlog))
+		if (ipa_endpoint_replenish_one(endpoint))
+			goto try_again_later;
+
+	return;
+
+try_again_later:
+	/* The last one didn't succeed, so fix the backlog */
+	backlog = atomic_inc_return(&endpoint->replenish_backlog);
+
+	/* Whenever a receive buffer transaction completes we'll try to
+	 * replenish again.  It's unlikely, but if we fail to supply even
+	 * one buffer, nothing will trigger another replenish attempt.
+	 * If this happens, schedule work to try again.
+	 */
+	gsi = &endpoint->ipa->gsi;
+	if (backlog == gsi_channel_trans_max(gsi, endpoint->channel_id))
+		schedule_delayed_work(&endpoint->replenish_work,
+				      msecs_to_jiffies(1));
+}
+
+static void ipa_endpoint_replenish_enable(struct ipa_endpoint *endpoint)
+{
+	struct gsi *gsi = &endpoint->ipa->gsi;
+	u32 max_backlog;
+	u32 saved;
+
+	endpoint->replenish_enabled = true;
+	while ((saved = atomic_xchg(&endpoint->replenish_saved, 0)))
+		atomic_add(saved, &endpoint->replenish_backlog);
+
+	/* Start replenishing if hardware currently has no buffers */
+	max_backlog = gsi_channel_trans_max(gsi, endpoint->channel_id);
+	if (atomic_read(&endpoint->replenish_backlog) == max_backlog) {
+		ipa_endpoint_replenish(endpoint, false);
+		return;
+	}
+}
+
+static void ipa_endpoint_replenish_disable(struct ipa_endpoint *endpoint)
+{
+	endpoint->replenish_enabled = false;
+}
+
+static void ipa_endpoint_replenish_work(struct work_struct *work)
+{
+	struct delayed_work *dwork = to_delayed_work(work);
+	struct ipa_endpoint *endpoint;
+
+	endpoint = container_of(dwork, struct ipa_endpoint, replenish_work);
+
+	ipa_endpoint_replenish(endpoint, false);
+}
+
+static bool ipa_endpoint_set_up(struct ipa_endpoint *endpoint)
+{
+	struct ipa *ipa = endpoint->ipa;
+
+	return ipa && (ipa->set_up & BIT(endpoint->endpoint_id));
+}
+
+static void ipa_endpoint_default_route_set(struct ipa *ipa,
+					   enum ipa_endpoint_id endpoint_id)
+{
+	u32 val;
+
+	/* ROUTE_DIS is 0 */
+	val = u32_encode_bits(endpoint_id, ROUTE_DEF_PIPE_FMASK);
+	val |= ROUTE_DEF_HDR_TABLE_FMASK;
+	val |= u32_encode_bits(0, ROUTE_DEF_HDR_OFST_FMASK);
+	val |= u32_encode_bits(endpoint_id, ROUTE_FRAG_DEF_PIPE_FMASK);
+	val |= ROUTE_DEF_RETAIN_HDR_FMASK;
+
+	iowrite32(val, ipa->reg_virt + IPA_REG_ROUTE_OFFSET);
+}
+/**
+ * ipa_endpoint_default_route_init() - Configure IPA default route
+ * @ipa:	IPA pointer
+ * @client:	Client to which exceptions should be directed
+ */
+void ipa_endpoint_default_route_setup(struct ipa_endpoint *endpoint)
+{
+	ipa_endpoint_default_route_set(endpoint->ipa, endpoint->endpoint_id);
+}
+
+/**
+ * ipa_endpoint_default_route_teardown() -
+ *			Inverse of ipa_endpoint_default_route_setup()
+ * @ipa:	IPA pointer
+ */
+void ipa_endpoint_default_route_teardown(struct ipa_endpoint *endpoint)
+{
+	ipa_endpoint_default_route_set(endpoint->ipa, 0);
+}
+
+/**
+ * ipa_endpoint_stop()- Stops a GSI channel in IPA
+ * @client:	Client whose endpoint should be stopped
+ *
+ * This function implements the sequence to stop a GSI channel
+ * in IPA. This function returns when the channel is is STOP state.
+ *
+ * Return value: 0 on success, negative otherwise
+ */
+int ipa_endpoint_stop(struct ipa_endpoint *endpoint)
+{
+	struct device *dev = &endpoint->ipa->pdev->dev;
+	size_t size = IPA_ENDPOINT_STOP_RX_SIZE;
+	struct gsi *gsi = &endpoint->ipa->gsi;
+	void *virt = NULL;
+	dma_addr_t addr;
+	int ret;
+	int i;
+
+	/* An RX endpoint might not stop right away.  In that case we issue
+	 * a small (1-byte) DMA command, delay for a bit (1-2 milliseconds),
+	 * and try again.  Allocate the DMA buffer in case this is needed.
+	 */
+	if (!endpoint->toward_ipa) {
+		virt = dma_alloc_coherent(dev, size, &addr, GFP_KERNEL);
+		if (!virt)
+			return -ENOMEM;
+	}
+
+	for (i = 0; i < IPA_ENDPOINT_STOP_RETRY_MAX; i++) {
+		ret = gsi_channel_stop(gsi, endpoint->channel_id);
+		if (ret != -EAGAIN)
+			break;
+
+		if (endpoint->toward_ipa)
+			continue;
+
+		/* Send a 1 byte 32-bit DMA task and try again after a delay */
+		ret = ipa_cmd_dma_task_32(endpoint->ipa, size, addr);
+		if (ret)
+			break;
+
+		usleep_range(USEC_PER_MSEC, 2 * USEC_PER_MSEC);
+	}
+	if (i >= IPA_ENDPOINT_STOP_RETRY_MAX)
+		ret = -EIO;
+
+	if (!endpoint->toward_ipa)
+		dma_free_coherent(dev, size, virt, addr);
+
+	return ret;
+}
+
+bool ipa_endpoint_enabled(struct ipa_endpoint *endpoint)
+{
+	return !!(endpoint->ipa->enabled & BIT(endpoint->endpoint_id));
+}
+
+int ipa_endpoint_enable_one(struct ipa_endpoint *endpoint)
+{
+	struct ipa *ipa = endpoint->ipa;
+	int ret;
+
+	if (WARN_ON(!ipa_endpoint_set_up(endpoint)))
+		return -EINVAL;
+
+	ret = gsi_channel_start(&ipa->gsi, endpoint->channel_id);
+	if (ret)
+		return ret;
+
+	ipa_interrupt_suspend_enable(ipa->interrupt, endpoint->endpoint_id);
+
+	if (!endpoint->toward_ipa)
+		ipa_endpoint_replenish_enable(endpoint);
+
+	ipa->enabled |= BIT(endpoint->endpoint_id);
+
+	return 0;
+}
+
+void ipa_endpoint_disable_one(struct ipa_endpoint *endpoint)
+{
+	struct ipa *ipa = endpoint->ipa;
+	int ret;
+
+	if (WARN_ON(!ipa_endpoint_enabled(endpoint)))
+		return;
+
+	if (!endpoint->toward_ipa)
+		ipa_endpoint_replenish_disable(endpoint);
+
+	ipa_interrupt_suspend_disable(ipa->interrupt, endpoint->endpoint_id);
+
+	ret = ipa_endpoint_stop(endpoint);
+	WARN(ret, "error %d attempting to stop endpoint %u\n", ret,
+	     endpoint->endpoint_id);
+
+	if (!ret)
+		endpoint->ipa->enabled &= ~BIT(endpoint->endpoint_id);
+}
+
+static bool ipa_endpoint_aggr_active(struct ipa_endpoint *endpoint)
+{
+	u32 mask = BIT(endpoint->endpoint_id);
+	struct ipa *ipa = endpoint->ipa;
+	u32 val;
+
+	val = ioread32(ipa->reg_virt + IPA_REG_STATE_AGGR_ACTIVE_OFFSET);
+
+	return !!(val & mask);
+}
+
+static void ipa_endpoint_force_close(struct ipa_endpoint *endpoint)
+{
+	u32 mask = BIT(endpoint->endpoint_id);
+	struct ipa *ipa = endpoint->ipa;
+	u32 val;
+
+	val = u32_encode_bits(mask, PIPE_BITMAP_FMASK);
+	iowrite32(val, ipa->reg_virt + IPA_REG_AGGR_FORCE_CLOSE_OFFSET);
+}
+
+/**
+ * ipa_endpoint_reset_rx_aggr() - Reset RX endpoint with aggregation active
+ * @endpoint:	Endpoint to be reset
+ *
+ * If aggregation is active on an RX endpoint when a reset is performed
+ * on its underlying GSI channel, a special sequence of actions must be
+ * taken to ensure the IPA pipeline is properly cleared.
+ *
+ * @Return:	0 if successful, or a negative error code
+ */
+static int ipa_endpoint_reset_rx_aggr(struct ipa_endpoint *endpoint)
+{
+	struct device *dev = &endpoint->ipa->pdev->dev;
+	struct ipa *ipa = endpoint->ipa;
+	bool endpoint_suspended = false;
+	struct gsi *gsi = &ipa->gsi;
+	dma_addr_t addr;
+	u32 len = 1;
+	void *virt;
+	int ret;
+	int i;
+
+	virt = kzalloc(len, GFP_KERNEL);
+	if (!virt)
+		return -ENOMEM;
+
+	addr = dma_map_single(dev, virt, len, DMA_FROM_DEVICE);
+	if (dma_mapping_error(dev, addr)) {
+		ret = -ENOMEM;
+		goto out_free_virt;
+	}
+
+	/* Force close aggregation before issuing the reset */
+	ipa_endpoint_force_close(endpoint);
+
+	/* Reset and reconfigure the channel with the doorbell engine
+	 * disabled.  Then poll until we know aggregation is no longer
+	 * active.  We'll re-enable the doorbell when we reset below.
+	 */
+	ret = gsi_channel_reset(gsi, endpoint->channel_id, false);
+	if (ret)
+		goto out_unmap_addr;
+
+	if (ipa_endpoint_init_ctrl(endpoint, false))
+		endpoint_suspended = true;
+
+	/* Start channel and do a 1 byte read */
+	ret = gsi_channel_start(gsi, endpoint->channel_id);
+	if (ret)
+		goto out_suspend_again;
+
+	ret = gsi_trans_read_byte(gsi, endpoint->channel_id, addr);
+	if (ret)
+		goto err_stop_channel;
+
+	/* Wait for aggregation to be closed on the channel */
+	for (i = 0; i < IPA_ENDPOINT_RESET_AGGR_RETRY_MAX; i++) {
+		if (!ipa_endpoint_aggr_active(endpoint))
+			break;
+		usleep_range(USEC_PER_MSEC, 2 * USEC_PER_MSEC);
+	}
+	WARN_ON(ipa_endpoint_aggr_active(endpoint));
+	gsi_trans_read_byte_done(gsi, endpoint->channel_id);
+
+	ret = ipa_endpoint_stop(endpoint);
+	if (ret)
+		goto out_suspend_again;
+
+	/* Finally, reset and reconfigure the channel again (this time with
+	 * the doorbell engine enabled).  Sleep for 1 millisecond to complete
+	 * the channel reset sequence.  Finish by suspending the channel
+	 * again (if necessary).
+	 */
+	ret = gsi_channel_reset(gsi, endpoint->channel_id, true);
+
+	usleep_range(USEC_PER_MSEC, 2 * USEC_PER_MSEC);
+
+	goto out_suspend_again;
+
+err_stop_channel:
+	ipa_endpoint_stop(endpoint);
+out_suspend_again:
+	if (endpoint_suspended)
+		(void)ipa_endpoint_init_ctrl(endpoint, true);
+out_unmap_addr:
+	dma_unmap_single(dev, addr, len, DMA_FROM_DEVICE);
+out_free_virt:
+	kfree(virt);
+
+	return ret;
+}
+
+static void ipa_endpoint_reset(struct ipa_endpoint *endpoint)
+{
+	u32 channel_id = endpoint->channel_id;
+	struct ipa *ipa = endpoint->ipa;
+	struct gsi *gsi = &ipa->gsi;
+	int ret;
+
+	/* For TX endpoints, or RX endpoints without aggregation active,
+	 * we only need to reset the underlying GSI channel.
+	 */
+	if (!endpoint->toward_ipa && endpoint->data->config.aggregation) {
+		if (ipa_endpoint_aggr_active(endpoint))
+			ret = ipa_endpoint_reset_rx_aggr(endpoint);
+		else
+			ret = gsi_channel_reset(gsi, channel_id, true);
+	} else {
+		ret = gsi_channel_reset(gsi, channel_id, true);
+	}
+	WARN(ret, "error %d attempting to reset channel %u\n", ret,
+	     endpoint->channel_id);
+}
+
+static bool ipa_endpoint_suspended(struct ipa_endpoint *endpoint)
+{
+	return !!(endpoint->ipa->suspended & BIT(endpoint->endpoint_id));
+}
+
+/**
+ * ipa_endpoint_suspend_aggr() - Emulate suspend interrupt
+ * @endpoint_id:	Endpoint on which to emulate a suspend
+ *
+ *  Emulate suspend IPA interrupt to unsuspend an endpoint suspended
+ *  with an open aggregation frame.  This is to work around a hardware
+ *  issue where the suspend interrupt will not be generated when it
+ *  should be.
+ */
+static void ipa_endpoint_suspend_aggr(struct ipa_endpoint *endpoint)
+{
+	struct ipa *ipa = endpoint->ipa;
+
+	/* Nothing to do if the endpoint doesn't have aggregation open */
+	if (!ipa_endpoint_aggr_active(endpoint))
+		return;
+
+	/* Force close aggregation */
+	ipa_endpoint_force_close(endpoint);
+
+	ipa_interrupt_simulate_suspend(ipa->interrupt);
+}
+
+void ipa_endpoint_suspend(struct ipa_endpoint *endpoint)
+{
+	struct gsi *gsi = &endpoint->ipa->gsi;
+
+	if (!ipa_endpoint_enabled(endpoint))
+		return;
+
+	if (!endpoint->toward_ipa) {
+		if (!ipa_endpoint_init_ctrl(endpoint, true))
+			return;
+
+		ipa_endpoint_replenish_disable(endpoint);
+
+		/* Due to a hardware bug, a client suspended with an open
+		 * aggregation frame will not generate a SUSPEND IPA interrupt.
+		 * We work around this by force-closing the aggregation frame,
+		 * then simulating the arrival of such an interrupt.
+		*/
+		if (endpoint->data->config.aggregation)
+			ipa_endpoint_suspend_aggr(endpoint);
+	}
+
+	gsi_channel_trans_quiesce(gsi, endpoint->channel_id);
+
+	endpoint->ipa->suspended |= BIT(endpoint->endpoint_id);
+}
+
+void ipa_endpoint_resume(struct ipa_endpoint *endpoint)
+{
+	if (!ipa_endpoint_suspended(endpoint))
+		return;
+
+	if (endpoint->toward_ipa)
+		ipa_endpoint_replenish_enable(endpoint);
+	else
+		WARN_ON(ipa_endpoint_init_ctrl(endpoint, false));
+
+	endpoint->ipa->suspended &= ~BIT(endpoint->endpoint_id);
+}
+
+static void ipa_endpoint_program(struct ipa_endpoint *endpoint)
+{
+	if (endpoint->toward_ipa) {
+		bool delay_mode = !!endpoint->data->config.tx.delay;
+
+		(void)ipa_endpoint_init_ctrl(endpoint, delay_mode);
+		ipa_endpoint_init_hdr_ext(endpoint);
+		ipa_endpoint_init_aggr(endpoint);
+		ipa_endpoint_init_deaggr(endpoint);
+		ipa_endpoint_init_seq(endpoint);
+	} else {
+		(void)ipa_endpoint_init_ctrl(endpoint, false);
+		ipa_endpoint_init_hdr_ext(endpoint);
+		ipa_endpoint_init_aggr(endpoint);
+	}
+	ipa_endpoint_init_cfg(endpoint);
+	ipa_endpoint_init_hdr(endpoint);
+	ipa_endpoint_init_hdr_metadata_mask(endpoint);
+	ipa_endpoint_init_mode(endpoint);
+	ipa_endpoint_status(endpoint);
+}
+
+static void ipa_endpoint_setup_one(struct ipa_endpoint *endpoint)
+{
+	struct gsi *gsi = &endpoint->ipa->gsi;
+	u32 channel_id = endpoint->channel_id;
+
+	/* Only AP endpoints get configured */
+	if (endpoint->ee_id != GSI_EE_AP)
+		return;
+
+	endpoint->trans_tre_max = gsi_channel_trans_tre_max(gsi, channel_id);
+	if (!endpoint->toward_ipa) {
+		endpoint->replenish_enabled = false;
+		atomic_set(&endpoint->replenish_saved,
+			   gsi_channel_trans_max(gsi, endpoint->channel_id));
+		atomic_set(&endpoint->replenish_backlog, 0);
+		INIT_DELAYED_WORK(&endpoint->replenish_work,
+				  ipa_endpoint_replenish_work);
+	}
+
+	ipa_endpoint_program(endpoint);
+
+	endpoint->ipa->set_up |= BIT(endpoint->endpoint_id);
+}
+
+static void ipa_endpoint_teardown_one(struct ipa_endpoint *endpoint)
+{
+	if (!endpoint->toward_ipa)
+		cancel_delayed_work_sync(&endpoint->replenish_work);
+
+	ipa_endpoint_reset(endpoint);
+
+	endpoint->ipa->set_up &= ~BIT(endpoint->endpoint_id);
+}
+
+void ipa_endpoint_setup(struct ipa *ipa)
+{
+	u32 initialized = ipa->initialized;
+
+	ipa->set_up = 0;
+	while (initialized) {
+		enum ipa_endpoint_id endpoint_id = __ffs(initialized);
+
+		initialized ^= BIT(endpoint_id);
+
+		ipa_endpoint_setup_one(&ipa->endpoint[endpoint_id]);
+	}
+}
+
+void ipa_endpoint_teardown(struct ipa *ipa)
+{
+	u32 set_up = ipa->set_up;
+
+	while (set_up) {
+		enum ipa_endpoint_id endpoint_id = __fls(set_up);
+
+		set_up ^= BIT(endpoint_id);
+
+		ipa_endpoint_teardown_one(&ipa->endpoint[endpoint_id]);
+	}
+}
+
+static int ipa_endpoint_init_one(struct ipa *ipa,
+				 const struct gsi_ipa_endpoint_data *data)
+{
+	struct ipa_endpoint *endpoint;
+
+	if (data->endpoint_id >= IPA_ENDPOINT_MAX)
+		return -EIO;
+	endpoint = &ipa->endpoint[data->endpoint_id];
+
+	if (data->ee_id == GSI_EE_AP)
+		ipa->endpoint_map[data->channel_id] = endpoint;
+
+	endpoint->ipa = ipa;
+	endpoint->ee_id = data->ee_id;
+	endpoint->channel_id = data->channel_id;
+	endpoint->endpoint_id = data->endpoint_id;
+	endpoint->toward_ipa = data->toward_ipa;
+	endpoint->data = &data->endpoint;
+
+	if (endpoint->data->support_flt)
+		ipa->filter_support |= BIT(endpoint->endpoint_id);
+
+	ipa->initialized |= BIT(endpoint->endpoint_id);
+
+	return 0;
+}
+
+void ipa_endpoint_exit_one(struct ipa_endpoint *endpoint)
+{
+	endpoint->ipa->initialized &= ~BIT(endpoint->endpoint_id);
+}
+
+int ipa_endpoint_init(struct ipa *ipa, u32 data_count,
+		      const struct gsi_ipa_endpoint_data *data)
+{
+	u32 initialized;
+	int ret;
+	u32 i;
+
+	ipa->initialized = 0;
+
+	ipa->filter_support = 0;
+	for (i = 0; i < data_count; i++) {
+		ret = ipa_endpoint_init_one(ipa, &data[i]);
+		if (ret)
+			goto err_endpoint_unwind;
+	}
+	dev_dbg(&ipa->pdev->dev, "initialized 0x%08x\n", ipa->initialized);
+
+	/* Verify the bitmap of endpoints that support filtering. */
+	dev_dbg(&ipa->pdev->dev, "filter_support 0x%08x\n",
+		ipa->filter_support);
+	if (!ipa->filter_support)
+		goto err_endpoint_unwind;
+	if (hweight32(ipa->filter_support) > IPA_SMEM_FLT_COUNT)
+		goto err_endpoint_unwind;
+
+	return 0;
+
+err_endpoint_unwind:
+	initialized = ipa->initialized;
+	while (initialized) {
+		enum ipa_endpoint_id endpoint_id = __fls(initialized);
+
+		initialized ^= BIT(endpoint_id);
+
+		ipa_endpoint_exit_one(&ipa->endpoint[endpoint_id]);
+	}
+
+	return ret;
+}
+
+void ipa_endpoint_exit(struct ipa *ipa)
+{
+	u32 initialized = ipa->initialized;
+
+	while (initialized) {
+		enum ipa_endpoint_id endpoint_id = __fls(initialized);
+
+		initialized ^= BIT(endpoint_id);
+
+		ipa_endpoint_exit_one(&ipa->endpoint[endpoint_id]);
+	}
+}
drivers/net/ipa/ipa_endpoint.h new file +97 -0

97 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#16 lore raw

PATCH v2 11/17 soc: qcom: ipa: immediate commands

In reply to Alex Elder

One TX endpoint (per EE) is used for issuing immediate commands to the IPA. These commands request activites beyond simple data transfers to be done by the IPA hardware. For example, the IPA is able to manage routing packets among endpoints, and immediate commands are used to configure tables used for that routing.

Immediate commands are built on top of GSI transactions. They are different from normal transfers (in that they use a special endpoint, and their "payload" is interpreted differently), so separate functions are used to issue immediate command transactions.

---
 drivers/net/ipa/ipa_cmd.c | 377 ++++++++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_cmd.h | 116 ++++++++++++
 2 files changed, 493 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_cmd.c
 create mode 100644 drivers/net/ipa/ipa_cmd.h
drivers/net/ipa/ipa_cmd.c new file +377 -0
new file mode 100644
index 000000000000..32b11941436d
@@ -0,0 +1,377 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+#include <linux/bitfield.h>
+
+#include "gsi.h"
+#include "gsi_trans.h"
+#include "ipa.h"
+#include "ipa_endpoint.h"
+#include "ipa_cmd.h"
+#include "ipa_mem.h"
+
+/**
+ * DOC:  IPA Immediate Commands
+ *
+ * The AP command TX endpoint is used to issue immediate commands to the IPA.
+ * An immediate command is generally used to request the IPA do something
+ * other than data transfer to another endpoint.
+ *
+ * Immediate commands are represented by GSI transactions just like other
+ * transfer requests, represented by a single GSI TRE.  Each immediate
+ * command has a well-defined format, having a payload of a known length.
+ * This allows the transfer element's length field to be used to hold an
+ * immediate command's opcode.  The payload for a command resides in DRAM
+ * and is described by a single scatterlist entry in its transaction.
+ * Commands do not require a transaction completion callback.  To commit
+ * an immediate command transaction, either gsi_trans_commit_command() or
+ * gsi_trans_commit_command_timeout() is used.
+ */
+
+#define IPA_GSI_DMA_TASK_TIMEOUT	15	/* milliseconds */
+
+/**
+ * __ipa_cmd_timeout() - Send an immediate command with timeout
+ * @ipa:	IPA structure
+ * @opcode:	Immediate command opcode (must not be IPA_CMD_NONE)
+ * @payload:	Pointer to command payload
+ * @size:	Size of payload
+ * @timeout:	Milliseconds to wait for completion (0 waits indefinitely)
+ *
+ * This common function implements ipa_cmd() and ipa_cmd_timeout().  It
+ * allocates, initializes, and commits a transaction for the immediate
+ * command.  The transaction is committed using gsi_trans_commit_command(),
+ * or if a non-zero timeout is supplied, gsi_trans_commit_command_timeout().
+ *
+ * @Return:	0 if successful, or a negative error code
+ */
+static int __ipa_cmd_timeout(struct ipa *ipa, enum ipa_cmd_opcode opcode,
+			     void *payload, size_t size, u32 timeout)
+{
+	struct ipa_endpoint *endpoint = ipa->command_endpoint;
+	struct gsi_trans *trans;
+	int ret;
+
+	/* assert(opcode != IPA_CMD_NONE) */
+	trans = gsi_channel_trans_alloc(&ipa->gsi, endpoint->channel_id, 1);
+	if (!trans)
+		return -EBUSY;
+
+	sg_init_one(trans->sgl, payload, size);
+
+	if (timeout)
+		ret = gsi_trans_commit_command_timeout(trans, opcode, timeout);
+	else
+		ret = gsi_trans_commit_command(trans, opcode);
+	if (ret)
+		goto err_trans_free;
+
+	return 0;
+
+err_trans_free:
+	gsi_trans_free(trans);
+
+	return ret;
+}
+
+static int
+ipa_cmd(struct ipa *ipa, enum ipa_cmd_opcode opcode, void *payload, size_t size)
+{
+	return __ipa_cmd_timeout(ipa, opcode, payload, size, 0);
+}
+
+static int ipa_cmd_timeout(struct ipa *ipa, enum ipa_cmd_opcode opcode,
+			   void *payload, size_t size)
+{
+	return __ipa_cmd_timeout(ipa, opcode, payload, size,
+				 IPA_GSI_DMA_TASK_TIMEOUT);
+}
+
+/* Field masks for ipa_imm_cmd_hw_hdr_init_local structure fields */
+#define IPA_CMD_HDR_INIT_FLAGS_TABLE_SIZE_FMASK	GENMASK(11, 0)
+#define IPA_CMD_HDR_INIT_FLAGS_HDR_ADDR_FMASK	GENMASK(27, 12)
+#define IPA_CMD_HDR_INIT_FLAGS_RESERVED_FMASK	GENMASK(28, 31)
+
+struct ipa_imm_cmd_hw_hdr_init_local {
+	u64 hdr_table_addr;
+	u32 flags;
+	u32 reserved;
+};
+
+/* Initialize header space in IPA-local memory */
+int ipa_cmd_hdr_init_local(struct ipa *ipa, u32 offset, u32 size)
+{
+	struct ipa_imm_cmd_hw_hdr_init_local *payload;
+	struct device *dev = &ipa->pdev->dev;
+	dma_addr_t addr;
+	void *virt;
+	u32 flags;
+	u32 max;
+	int ret;
+
+	if (size > field_max(IPA_CMD_HDR_INIT_FLAGS_TABLE_SIZE_FMASK))
+		return -EINVAL;
+
+	max = field_max(IPA_CMD_HDR_INIT_FLAGS_HDR_ADDR_FMASK);
+	if (offset > max || ipa->shared_offset > max - offset)
+		return -EINVAL;
+	offset += ipa->shared_offset;
+
+	/* With this command we tell the IPA where in its local memory the
+	 * header tables reside.  We also supply a (host) buffer whose
+	 * content is copied via DMA into that table space.  We just want
+	 * to zero fill it, so a zeroed DMA buffer is all that's required
+	 * The IPA owns the table, but the AP must initialize it.
+	 */
+	virt = dma_alloc_coherent(dev, size, &addr, GFP_KERNEL);
+	if (!virt)
+		return -ENOMEM;
+
+	payload = kzalloc(sizeof(*payload), GFP_KERNEL);
+	if (!payload) {
+		ret = -ENOMEM;
+		goto out_dma_free;
+	}
+
+	payload->hdr_table_addr = addr;
+	flags = u32_encode_bits(size, IPA_CMD_HDR_INIT_FLAGS_TABLE_SIZE_FMASK);
+	flags |= u32_encode_bits(offset, IPA_CMD_HDR_INIT_FLAGS_HDR_ADDR_FMASK);
+	payload->flags = flags;
+
+	ret = ipa_cmd(ipa, IPA_CMD_HDR_INIT_LOCAL, payload, sizeof(*payload));
+
+	kfree(payload);
+out_dma_free:
+	dma_free_coherent(dev, size, virt, addr);
+
+	return ret;
+}
+
+enum ipahal_pipeline_clear_option {
+	IPAHAL_HPS_CLEAR		= 0,
+	IPAHAL_SRC_GRP_CLEAR		= 1,
+	IPAHAL_FULL_PIPELINE_CLEAR	= 2,
+};
+
+/* Field masks for ipa_imm_cmd_hw_dma_shared_mem structure fields */
+#define IPA_CMD_DMA_SHARED_FLAGS_DIRECTION_FMASK	GENMASK(0, 0)
+#define IPA_CMD_DMA_SHARED_FLAGS_SKIP_CLEAR_FMASK	GENMASK(1, 1)
+#define IPA_CMD_DMA_SHARED_FLAGS_CLEAR_OPTIONS_FMASK	GENMASK(3, 2)
+
+struct ipa_imm_cmd_hw_dma_shared_mem {
+	u16 sw_reserved;
+	u16 size;
+	u16 local_addr;
+	u16 flags;
+	u64 system_addr;
+};
+
+/* Use a DMA command to zero a block of memory */
+int ipa_cmd_smem_dma_zero(struct ipa *ipa, u32 offset, u32 size)
+{
+	struct ipa_imm_cmd_hw_dma_shared_mem *payload;
+	struct device *dev = &ipa->pdev->dev;
+	dma_addr_t addr;
+	void *virt;
+	u32 val;
+	int ret;
+
+	/* size must be non-zero, and must fit in a 16 bit field */
+	if (!size || size > U16_MAX)
+		return -EINVAL;
+
+	/* offset must fit in a 16 bit local_addr field */
+	if (offset > U16_MAX || ipa->shared_offset > U16_MAX - offset)
+		return -EINVAL;
+	offset += ipa->shared_offset;
+
+	/* A zero-filled buffer of the right size is all that's required */
+	virt = dma_alloc_coherent(dev, size, &addr, GFP_KERNEL);
+	if (!virt)
+		return -ENOMEM;
+
+	payload = kzalloc(sizeof(*payload), GFP_KERNEL);
+	if (!payload) {
+		ret = -ENOMEM;
+		goto out_dma_free;
+	}
+
+	payload->size = size;
+	payload->local_addr = offset;
+	/* direction: 0 = write to IPA; skip clear: 0 = don't wait */
+	val = u16_encode_bits(IPAHAL_HPS_CLEAR,
+			      IPA_CMD_DMA_SHARED_FLAGS_CLEAR_OPTIONS_FMASK);
+	payload->flags = val;
+	payload->system_addr = addr;
+
+	ret = ipa_cmd(ipa, IPA_CMD_DMA_SHARED_MEM, payload, sizeof(*payload));
+
+	kfree(payload);
+out_dma_free:
+	dma_free_coherent(dev, size, virt, addr);
+
+	return ret;
+}
+
+/* Field masks for ipa_imm_cmd_hw_ip_fltrt_init structure fields */
+#define IPA_CMD_IP_FLTRT_FLAGS_HASH_SIZE_FMASK	GENMASK_ULL(11, 0)
+#define IPA_CMD_IP_FLTRT_FLAGS_HASH_ADDR_FMASK	GENMASK_ULL(27, 12)
+#define IPA_CMD_IP_FLTRT_FLAGS_NHASH_SIZE_FMASK	GENMASK_ULL(39, 28)
+#define IPA_CMD_IP_FLTRT_FLAGS_NHASH_ADDR_FMASK	GENMASK_ULL(55, 40)
+
+struct ipa_imm_cmd_hw_ip_fltrt_init {
+	u64 hash_rules_addr;
+	u64 flags;
+	u64 nhash_rules_addr;
+};
+
+/* Configure a routing or filter table, for IPv4 or IPv6 */
+static int ipa_cmd_table_config(struct ipa *ipa, enum ipa_cmd_opcode opcode,
+				dma_addr_t addr, size_t size, u32 hash_offset,
+				u32 nhash_offset)
+{
+	struct ipa_imm_cmd_hw_ip_fltrt_init *payload;
+	u64 val;
+	u32 max;
+	int ret;
+
+	if (size > field_max(IPA_CMD_IP_FLTRT_FLAGS_HASH_SIZE_FMASK))
+		return -EINVAL;
+	if (size > field_max(IPA_CMD_IP_FLTRT_FLAGS_NHASH_SIZE_FMASK))
+		return -EINVAL;
+
+	max = field_max(IPA_CMD_IP_FLTRT_FLAGS_HASH_ADDR_FMASK);
+	if (hash_offset > max || ipa->shared_offset > max - hash_offset)
+		return -EINVAL;
+	hash_offset += ipa->shared_offset;
+
+	max = field_max(IPA_CMD_IP_FLTRT_FLAGS_NHASH_ADDR_FMASK);
+	if (nhash_offset > max || ipa->shared_offset > max - nhash_offset)
+		return -EINVAL;
+	nhash_offset += ipa->shared_offset;
+
+	payload = kzalloc(sizeof(*payload), GFP_KERNEL);
+	if (!payload)
+		return -ENOMEM;
+
+	payload->hash_rules_addr = addr;
+	val = u64_encode_bits(size, IPA_CMD_IP_FLTRT_FLAGS_HASH_SIZE_FMASK);
+	val |= u64_encode_bits(hash_offset,
+			       IPA_CMD_IP_FLTRT_FLAGS_HASH_ADDR_FMASK);
+	val |= u64_encode_bits(size, IPA_CMD_IP_FLTRT_FLAGS_NHASH_SIZE_FMASK);
+	val |= u64_encode_bits(nhash_offset,
+			       IPA_CMD_IP_FLTRT_FLAGS_NHASH_ADDR_FMASK);
+	payload->flags = val;
+	payload->nhash_rules_addr = addr;
+
+	ret = ipa_cmd(ipa, opcode, payload, sizeof(*payload));
+
+	kfree(payload);
+
+	return ret;
+}
+
+/* Configure IPv4 routing table */
+int ipa_cmd_route_config_ipv4(struct ipa *ipa, size_t size)
+{
+	enum ipa_cmd_opcode opcode = IPA_CMD_IP_V4_ROUTING_INIT;
+	u32 nhash_offset = IPA_SMEM_V4_RT_NHASH_OFFSET;
+	u32 hash_offset = IPA_SMEM_V4_RT_HASH_OFFSET;
+	dma_addr_t addr = ipa->route_addr;
+
+	return ipa_cmd_table_config(ipa, opcode, addr, size, hash_offset,
+				    nhash_offset);
+}
+
+/* Configure IPv6 routing table */
+int ipa_cmd_route_config_ipv6(struct ipa *ipa, size_t size)
+{
+	enum ipa_cmd_opcode opcode = IPA_CMD_IP_V6_ROUTING_INIT;
+	u32 nhash_offset = IPA_SMEM_V6_RT_NHASH_OFFSET;
+	u32 hash_offset = IPA_SMEM_V6_RT_HASH_OFFSET;
+	dma_addr_t addr = ipa->route_addr;
+
+	return ipa_cmd_table_config(ipa, opcode, addr, size, hash_offset,
+				    nhash_offset);
+}
+
+/* Configure IPv4 filter table */
+int ipa_cmd_filter_config_ipv4(struct ipa *ipa, size_t size)
+{
+	enum ipa_cmd_opcode opcode = IPA_CMD_IP_V4_FILTER_INIT;
+	u32 nhash_offset = IPA_SMEM_V4_FLT_NHASH_OFFSET;
+	u32 hash_offset = IPA_SMEM_V4_FLT_HASH_OFFSET;
+	dma_addr_t addr = ipa->filter_addr;
+
+	return ipa_cmd_table_config(ipa, opcode, addr, size, hash_offset,
+				    nhash_offset);
+}
+
+/* Configure IPv6 filter table */
+int ipa_cmd_filter_config_ipv6(struct ipa *ipa, size_t size)
+{
+	enum ipa_cmd_opcode opcode = IPA_CMD_IP_V6_FILTER_INIT;
+	u32 nhash_offset = IPA_SMEM_V6_FLT_NHASH_OFFSET;
+	u32 hash_offset = IPA_SMEM_V6_FLT_HASH_OFFSET;
+	dma_addr_t addr = ipa->filter_addr;
+
+	return ipa_cmd_table_config(ipa, opcode, addr, size, hash_offset,
+				    nhash_offset);
+}
+
+/* Field masks for ipa_imm_cmd_hw_dma_task_32b_addr structure fields */
+#define IPA_CMD_DMA32_TASK_SW_RSVD_FMASK	GENMASK(10, 0)
+#define IPA_CMD_DMA32_TASK_CMPLT_FMASK		GENMASK(11, 11)
+#define IPA_CMD_DMA32_TASK_EOF_FMASK		GENMASK(12, 12)
+#define IPA_CMD_DMA32_TASK_FLSH_FMASK		GENMASK(13, 13)
+#define IPA_CMD_DMA32_TASK_LOCK_FMASK		GENMASK(14, 14)
+#define IPA_CMD_DMA32_TASK_UNLOCK_FMASK		GENMASK(15, 15)
+#define IPA_CMD_DMA32_SIZE1_FMASK		GENMASK(31, 16)
+#define IPA_CMD_DMA32_PACKET_SIZE_FMASK		GENMASK(15, 0)
+
+struct ipa_imm_cmd_hw_dma_task_32b_addr {
+	u32 size1_flags;
+	u32 addr1;
+	u32 packet_size;
+	u32 reserved;
+};
+
+/* Use a 32-bit DMA command to zero a block of memory */
+int ipa_cmd_dma_task_32(struct ipa *ipa, size_t size, dma_addr_t addr)
+{
+	struct ipa_imm_cmd_hw_dma_task_32b_addr *payload;
+	u32 size1_flags;
+	int ret;
+
+	if (size > field_max(IPA_CMD_DMA32_SIZE1_FMASK))
+		return -EINVAL;
+	if (size > field_max(IPA_CMD_DMA32_PACKET_SIZE_FMASK))
+		return -EINVAL;
+
+	payload = kzalloc(sizeof(*payload), GFP_KERNEL);
+	if (!payload)
+		return -ENOMEM;
+
+	/* complete: 0 = don't interrupt; eof: 0 = don't assert eot */
+	size1_flags = IPA_CMD_DMA32_TASK_FLSH_FMASK;
+	/* lock: 0 = don't lock endpoint; unlock: 0 = don't unlock */
+	size1_flags |= u32_encode_bits(size, IPA_CMD_DMA32_SIZE1_FMASK);
+
+	payload->size1_flags = size1_flags;
+	payload->addr1 = addr;
+	payload->packet_size =
+			u32_encode_bits(size, IPA_CMD_DMA32_PACKET_SIZE_FMASK);
+
+	ret = ipa_cmd_timeout(ipa, IPA_CMD_DMA_TASK_32B_ADDR, payload,
+			      sizeof(payload));
+
+	kfree(payload);
+
+	return ret;
+}
drivers/net/ipa/ipa_cmd.h new file +116 -0

116 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#17 lore raw

PATCH v2 12/17 soc: qcom: ipa: IPA network device and microcontroller

In reply to Alex Elder

This patch includes the code that implements a Linux network device, using one TX and one RX IPA endpoint. It is used to implement the network device representing the modem and its connection to wireless networks. There are only a few things that are really modem-specific though, and they aren't clearly called out here. Such distinctions will be made clearer if we wish to support a network device for anything other than the modem.

Sort of unrelated, this patch also includes the code supporting the microcontroller CPU present on the IPA. The microcontroller can be used to implement special handling of packets, but at this time we don't support that. Still, it is a component that needs to be initialized, and in the event of a crash we need to do some synchronization between the AP and the microcontroller.

---
 drivers/net/ipa/ipa_netdev.c | 251 +++++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_netdev.h |  24 ++++
 drivers/net/ipa/ipa_uc.c     | 208 +++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_uc.h     |  32 +++++
 4 files changed, 515 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_netdev.c
 create mode 100644 drivers/net/ipa/ipa_netdev.h
 create mode 100644 drivers/net/ipa/ipa_uc.c
 create mode 100644 drivers/net/ipa/ipa_uc.h
drivers/net/ipa/ipa_netdev.c new file +251 -0
new file mode 100644
index 000000000000..19c73c4da02b
@@ -0,0 +1,251 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2014-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+
+/* Modem Transport Network Driver. */
+
+#include <linux/errno.h>
+#include <linux/if_arp.h>
+#include <linux/netdevice.h>
+#include <linux/skbuff.h>
+#include <linux/if_rmnet.h>
+
+#include "ipa.h"
+#include "ipa_data.h"
+#include "ipa_endpoint.h"
+#include "ipa_mem.h"
+#include "ipa_netdev.h"
+#include "ipa_qmi.h"
+
+#define IPA_NETDEV_NAME		"rmnet_ipa%d"
+
+#define TAILROOM		0	/* for padding by mux layer */
+
+#define IPA_NETDEV_TIMEOUT	10	/* seconds */
+
+/** struct ipa_priv - IPA network device private data */
+struct ipa_priv {
+	struct ipa_endpoint *tx_endpoint;
+	struct ipa_endpoint *rx_endpoint;
+};
+
+/** ipa_netdev_open() - Opens the modem network interface */
+static int ipa_netdev_open(struct net_device *netdev)
+{
+	struct ipa_priv *priv = netdev_priv(netdev);
+	int ret;
+
+	ret = ipa_endpoint_enable_one(priv->tx_endpoint);
+	if (ret)
+		return ret;
+	ret = ipa_endpoint_enable_one(priv->rx_endpoint);
+	if (ret)
+		goto err_disable_tx;
+
+	netif_start_queue(netdev);
+
+	return 0;
+
+err_disable_tx:
+	ipa_endpoint_disable_one(priv->tx_endpoint);
+
+	return ret;
+}
+
+/** ipa_netdev_stop() - Stops the modem network interface. */
+static int ipa_netdev_stop(struct net_device *netdev)
+{
+	struct ipa_priv *priv = netdev_priv(netdev);
+
+	netif_stop_queue(netdev);
+
+	ipa_endpoint_disable_one(priv->rx_endpoint);
+	ipa_endpoint_disable_one(priv->tx_endpoint);
+
+	return 0;
+}
+
+/** ipa_netdev_xmit() - Transmits an skb.
+ * @skb: skb to be transmitted
+ * @dev: network device
+ *
+ * Return codes:
+ * NETDEV_TX_OK: Success
+ * NETDEV_TX_BUSY: Error while transmitting the skb. Try again later
+ */
+static int ipa_netdev_xmit(struct sk_buff *skb, struct net_device *netdev)
+{
+	struct net_device_stats *stats = &netdev->stats;
+	struct ipa_priv *priv = netdev_priv(netdev);
+	struct ipa_endpoint *endpoint;
+	u32 skb_len = skb->len;
+
+	if (!skb_len)
+		goto err_drop;
+
+	endpoint = priv->tx_endpoint;
+	if (endpoint->data->config.qmap && skb->protocol != htons(ETH_P_MAP))
+		goto err_drop;
+
+	if (ipa_endpoint_skb_tx(endpoint, skb))
+		return NETDEV_TX_BUSY;
+
+	stats->tx_packets++;
+	stats->tx_bytes += skb_len;
+
+	return NETDEV_TX_OK;
+
+err_drop:
+	dev_kfree_skb_any(skb);
+	stats->tx_dropped++;
+
+	return NETDEV_TX_OK;
+}
+
+void ipa_netdev_skb_rx(struct net_device *netdev, struct sk_buff *skb)
+{
+	struct net_device_stats *stats = &netdev->stats;
+
+	if (skb) {
+		skb->dev = netdev;
+		skb->protocol = htons(ETH_P_MAP);
+		stats->rx_packets++;
+		stats->rx_bytes += skb->len;
+
+		(void)netif_receive_skb(skb);
+	} else {
+		stats->rx_dropped++;
+	}
+}
+
+static const struct net_device_ops ipa_netdev_ops = {
+	.ndo_open	= ipa_netdev_open,
+	.ndo_stop	= ipa_netdev_stop,
+	.ndo_start_xmit	= ipa_netdev_xmit,
+};
+
+/** netdev_setup() - netdev setup function  */
+static void netdev_setup(struct net_device *netdev)
+{
+	netdev->netdev_ops = &ipa_netdev_ops;
+	ether_setup(netdev);
+	/* No header ops (override value set by ether_setup()) */
+	netdev->header_ops = NULL;
+	netdev->type = ARPHRD_RAWIP;
+	netdev->hard_header_len = 0;
+	netdev->max_mtu = IPA_MTU;
+	netdev->mtu = netdev->max_mtu;
+	netdev->addr_len = 0;
+	netdev->flags &= ~(IFF_BROADCAST | IFF_MULTICAST);
+	/* The endpoint is configured for QMAP */
+	netdev->needed_headroom = sizeof(struct rmnet_map_header);
+	netdev->needed_tailroom = TAILROOM;
+	netdev->watchdog_timeo = IPA_NETDEV_TIMEOUT * HZ;
+	netdev->hw_features = NETIF_F_SG;
+}
+
+/** ipa_netdev_suspend() - suspend callback for runtime_pm
+ * @dev: pointer to device
+ *
+ * This callback will be invoked by the runtime_pm framework when an AP suspend
+ * operation is invoked, usually by pressing a suspend button.
+ *
+ * Returns -EAGAIN to runtime_pm framework in case there are pending packets
+ * in the Tx queue. This will postpone the suspend operation until all the
+ * pending packets will be transmitted.
+ *
+ * In case there are no packets to send, releases the WWAN0_PROD entity.
+ * As an outcome, the number of IPA active clients should be decremented
+ * until IPA clocks can be gated.
+ */
+void ipa_netdev_suspend(struct net_device *netdev)
+{
+	struct ipa_priv *priv = netdev_priv(netdev);
+
+	netif_stop_queue(netdev);
+
+	ipa_endpoint_suspend(priv->tx_endpoint);
+	ipa_endpoint_suspend(priv->rx_endpoint);
+}
+
+/** ipa_netdev_resume() - resume callback for runtime_pm
+ * @dev: pointer to device
+ *
+ * This callback will be invoked by the runtime_pm framework when an AP resume
+ * operation is invoked.
+ *
+ * Enables the network interface queue and returns success to the
+ * runtime_pm framework.
+ */
+void ipa_netdev_resume(struct net_device *netdev)
+{
+	struct ipa_priv *priv = netdev_priv(netdev);
+
+	ipa_endpoint_resume(priv->rx_endpoint);
+	ipa_endpoint_resume(priv->tx_endpoint);
+
+	netif_wake_queue(netdev);
+}
+
+struct net_device *ipa_netdev_setup(struct ipa *ipa,
+				    struct ipa_endpoint *rx_endpoint,
+				    struct ipa_endpoint *tx_endpoint)
+{
+	struct net_device *netdev;
+	struct ipa_priv *priv;
+	int ret;
+
+	/* Zero modem shared memory before we begin */
+	ret = ipa_smem_zero_modem(ipa);
+	if (ret)
+		return ERR_PTR(ret);
+
+	/* Start QMI communication with the modem */
+	ret = ipa_qmi_setup(ipa);
+	if (ret)
+		return ERR_PTR(ret);
+
+	netdev = alloc_netdev(sizeof(struct ipa_priv), IPA_NETDEV_NAME,
+			      NET_NAME_UNKNOWN, netdev_setup);
+	if (!netdev) {
+		ret = -ENOMEM;
+		goto err_qmi_exit;
+	}
+
+	rx_endpoint->netdev = netdev;
+	tx_endpoint->netdev = netdev;
+
+	priv = netdev_priv(netdev);
+	priv->tx_endpoint = tx_endpoint;
+	priv->rx_endpoint = rx_endpoint;
+
+	ret = register_netdev(netdev);
+	if (ret)
+		goto err_free_netdev;
+
+	return netdev;
+
+err_free_netdev:
+	free_netdev(netdev);
+err_qmi_exit:
+	ipa_qmi_teardown(ipa);
+
+	return ERR_PTR(ret);
+}
+
+void ipa_netdev_teardown(struct net_device *netdev)
+{
+	struct ipa_priv *priv = netdev_priv(netdev);
+	struct ipa *ipa = priv->tx_endpoint->ipa;
+
+	if (!netif_queue_stopped(netdev))
+		(void)ipa_netdev_stop(netdev);
+
+	unregister_netdev(netdev);
+
+	free_netdev(netdev);
+
+	ipa_qmi_teardown(ipa);
+}
drivers/net/ipa/ipa_netdev.h new file +24 -0
new file mode 100644
index 000000000000..8ab1e8ea0b4a
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+#ifndef _IPA_NETDEV_H_
+#define _IPA_NETDEV_H_
+
+struct ipa;
+struct ipa_endpoint;
+struct net_device;
+struct sk_buff;
+
+struct net_device *ipa_netdev_setup(struct ipa *ipa,
+				    struct ipa_endpoint *rx_endpoint,
+				    struct ipa_endpoint *tx_endpoint);
+void ipa_netdev_teardown(struct net_device *netdev);
+
+void ipa_netdev_skb_rx(struct net_device *netdev, struct sk_buff *skb);
+
+void ipa_netdev_suspend(struct net_device *netdev);
+void ipa_netdev_resume(struct net_device *netdev);
+
+#endif /* _IPA_NETDEV_H_ */
drivers/net/ipa/ipa_uc.c new file +208 -0

208 lines not on this page · raw message

drivers/net/ipa/ipa_uc.h new file +32 -0
new file mode 100644
index 000000000000..c258cb6e1161
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2019 Linaro Ltd.
+ */
+#ifndef _IPA_UC_H_
+#define _IPA_UC_H_
+
+struct ipa;
+
+/**
+ * ipa_uc_setup() - set up the IPA microcontroller subsystem
+ * @ipa:	IPA pointer
+ */
+void ipa_uc_setup(struct ipa *ipa);
+
+/**
+ * ipa_uc_teardown() - inverse of ipa_uc_setup()
+ * @ipa:	IPA pointer
+ */
+void ipa_uc_teardown(struct ipa *ipa);
+
+/**
+ * ipa_uc_panic_notifier()
+ * @ipa:	IPA pointer
+ *
+ * Notifier function called when the system crashes, to inform the
+ * microcontroller of the event.
+ */
+void ipa_uc_panic_notifier(struct ipa *ipa);
+
+#endif /* _IPA_UC_H_ */
Signature · 1 line
2.20.1
author
#18 lore raw

PATCH v2 13/17 soc: qcom: ipa: AP/modem communications

In reply to Alex Elder

This patch implements two forms of out-of-band communication between the AP and modem.

- QMI is a mechanism that allows clients running on the AP interact with services running on the modem (and vice-versa). The AP IPA driver uses QMI to communicate with the corresponding IPA driver resident on the modem, to agree on parameters used with the IPA hardware and to ensure both sides are ready before entering operational mode.

- SMP2P is a more primitive mechanism available for the modem and AP to communicate with each other. It provides a means for either the AP or modem to interrupt the other, and furthermore, to provide 32 bits worth of information. The IPA driver uses SMP2P to tell the modem what the state of the IPA clock was in the event of a crash. This allows the modem to safely access the IPA hardware (or avoid doing so) when a crash occurs, for example, to access information within the IPA hardware.

---
 drivers/net/ipa/ipa_qmi.c     | 402 +++++++++++++++++++++++
 drivers/net/ipa/ipa_qmi.h     |  35 ++
 drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++++++++++++++++++++++++++++++
 drivers/net/ipa/ipa_qmi_msg.h | 238 ++++++++++++++
 drivers/net/ipa/ipa_smp2p.c   | 304 ++++++++++++++++++
 drivers/net/ipa/ipa_smp2p.h   |  47 +++
 6 files changed, 1609 insertions(+)
 create mode 100644 drivers/net/ipa/ipa_qmi.c
 create mode 100644 drivers/net/ipa/ipa_qmi.h
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.c
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.h
 create mode 100644 drivers/net/ipa/ipa_smp2p.c
 create mode 100644 drivers/net/ipa/ipa_smp2p.h
drivers/net/ipa/ipa_qmi.c new file +402 -0
new file mode 100644
index 000000000000..e94437508f6c
@@ -0,0 +1,402 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/string.h>
+#include <linux/slab.h>
+#include <linux/qrtr.h>
+#include <linux/soc/qcom/qmi.h>
+
+#include "ipa.h"
+#include "ipa_endpoint.h"
+#include "ipa_mem.h"
+#include "ipa_qmi_msg.h"
+
+#define QMI_INIT_DRIVER_TIMEOUT	60000	/* A minute in milliseconds */
+
+/**
+ * DOC: AP/Modem QMI Handshake
+ *
+ * The AP and modem perform a "handshake" at initialization time to ensure
+ * each side knows the other side is ready.  Two QMI handles (endpoints) are
+ * used for this; one provides service on the modem for AP requests, and the
+ * other is on the AP to service modem requests (and to supply an indication
+ * from the AP).
+ *
+ * The QMI service on the modem expects to receive an INIT_DRIVER request from
+ * the AP, which contains parameters used by the modem during initialization.
+ * The AP sends this request using the client handle as soon as it is knows
+ * the modem side service is available.  The modem responds to this request
+ * immediately.
+ *
+ * When the modem learns the AP service is available, it is able to
+ * communicate its status to the AP.  The modem uses this to tell
+ * the AP when it is ready to receive an indication, sending an
+ * INDICATION_REGISTER request to the handle served by the AP.  This
+ * is independent of the modem's initialization of its driver.
+ *
+ * When the modem has completed the driver initialization requested by the
+ * AP, it sends a DRIVER_INIT_COMPLETE request to the AP.   This request
+ * could arrive at the AP either before or after the INDICATION_REGISTER
+ * request.
+ *
+ * The final step in the handshake occurs after the AP has received both
+ * requests from the modem.  The AP completes the handshake by sending an
+ * INIT_COMPLETE_IND indication message to the modem.
+ */
+
+#define IPA_HOST_SERVICE_SVC_ID		0x31
+#define IPA_HOST_SVC_VERS		1
+#define IPA_HOST_SERVICE_INS_ID		1
+
+#define IPA_MODEM_SERVICE_SVC_ID	0x31
+#define IPA_MODEM_SERVICE_INS_ID	2
+#define IPA_MODEM_SVC_VERS		1
+
+/* Send an INIT_COMPLETE_IND indication message to the modem */
+static int ipa_send_master_driver_init_complete_ind(struct qmi_handle *qmi,
+						    struct sockaddr_qrtr *sq)
+{
+	struct ipa_init_complete_ind ind = { };
+
+	ind.status.result = QMI_RESULT_SUCCESS_V01;
+	ind.status.error = QMI_ERR_NONE_V01;
+
+	return qmi_send_indication(qmi, sq, IPA_QMI_INIT_COMPLETE_IND,
+				   IPA_QMI_INIT_COMPLETE_IND_SZ,
+				   ipa_init_complete_ind_ei, &ind);
+}
+
+/* This function is called to determine whether to complete the handshake by
+ * sending an INIT_COMPLETE_IND indication message to the modem.  The
+ * "init_driver" parameter is false when we've received an INDICATION_REGISTER
+ * request message from the modem, or true when we've received the response
+ * from the INIT_DRIVER request message we send.  If this function decides the
+ * message should be sent, it calls ipa_send_master_driver_init_complete_ind()
+ * to send it.
+ */
+static void ipa_handshake_complete(struct qmi_handle *qmi,
+				   struct sockaddr_qrtr *sq, bool init_driver)
+{
+	struct ipa *ipa;
+	bool send_it;
+	int ret;
+
+	if (init_driver) {
+		ipa = container_of(qmi, struct ipa, qmi.client_handle);
+		ipa->qmi.init_driver_response_received = true;
+		send_it = !!ipa->qmi.indication_register_received;
+	} else {
+		ipa = container_of(qmi, struct ipa, qmi.server_handle);
+		ipa->qmi.indication_register_received = 1;
+		send_it = !!ipa->qmi.init_driver_response_received;
+	}
+	if (!send_it)
+		return;
+
+	ret = ipa_send_master_driver_init_complete_ind(qmi, sq);
+	WARN(ret, "error %d sending init complete indication\n", ret);
+}
+
+/* Callback function to handle an INDICATION_REGISTER request message from the
+ * modem.  This informs the AP that the modem is now ready to receive the
+ * INIT_COMPLETE_IND indication message.
+ */
+static void ipa_indication_register_fn(struct qmi_handle *qmi,
+				       struct sockaddr_qrtr *sq,
+				       struct qmi_txn *txn,
+				       const void *decoded)
+{
+	struct ipa_indication_register_rsp rsp = { };
+	int ret;
+
+	rsp.rsp.result = QMI_RESULT_SUCCESS_V01;
+	rsp.rsp.error = QMI_ERR_NONE_V01;
+
+	ret = qmi_send_response(qmi, sq, txn, IPA_QMI_INDICATION_REGISTER,
+				IPA_QMI_INDICATION_REGISTER_RSP_SZ,
+				ipa_indication_register_rsp_ei, &rsp);
+	if (!WARN(ret, "error %d sending response\n", ret))
+		ipa_handshake_complete(qmi, sq, false);
+}
+
+/* Callback function to handle a DRIVER_INIT_COMPLETE request message from the
+ * modem.  This informs the AP that the modem has completed the initializion
+ * of its driver.
+ */
+static void ipa_driver_init_complete_fn(struct qmi_handle *qmi,
+					struct sockaddr_qrtr *sq,
+					struct qmi_txn *txn,
+					const void *decoded)
+{
+	struct ipa_driver_init_complete_rsp rsp = { };
+	int ret;
+
+	rsp.rsp.result = QMI_RESULT_SUCCESS_V01;
+	rsp.rsp.error = QMI_ERR_NONE_V01;
+
+	ret = qmi_send_response(qmi, sq, txn, IPA_QMI_DRIVER_INIT_COMPLETE,
+				IPA_QMI_DRIVER_INIT_COMPLETE_RSP_SZ,
+				ipa_driver_init_complete_rsp_ei, &rsp);
+
+	WARN(ret, "error %d sending response\n", ret);
+}
+
+/* The server handles two request message types sent by the modem. */
+static struct qmi_msg_handler ipa_server_msg_handlers[] = {
+	{
+		.type		= QMI_REQUEST,
+		.msg_id		= IPA_QMI_INDICATION_REGISTER,
+		.ei		= ipa_indication_register_req_ei,
+		.decoded_size	= IPA_QMI_INDICATION_REGISTER_REQ_SZ,
+		.fn		= ipa_indication_register_fn,
+	},
+	{
+		.type		= QMI_REQUEST,
+		.msg_id		= IPA_QMI_DRIVER_INIT_COMPLETE,
+		.ei		= ipa_driver_init_complete_req_ei,
+		.decoded_size	= IPA_QMI_DRIVER_INIT_COMPLETE_REQ_SZ,
+		.fn		= ipa_driver_init_complete_fn,
+	},
+};
+
+/* Callback function to handle an IPA_QMI_INIT_DRIVER response message from
+ * the modem.  This only acknowledges that the modem received the request.
+ * The modem will eventually report that it has completed its modem
+ * initialization by sending a IPA_QMI_DRIVER_INIT_COMPLETE request.
+ */
+static void ipa_init_driver_rsp_fn(struct qmi_handle *qmi,
+				   struct sockaddr_qrtr *sq,
+				   struct qmi_txn *txn,
+				   const void *decoded)
+{
+	txn->result = 0;	/* IPA_QMI_INIT_DRIVER request was successful */
+	complete(&txn->completion);
+
+	ipa_handshake_complete(qmi, sq, true);
+}
+
+/* The client handles one response message type sent by the modem. */
+static struct qmi_msg_handler ipa_client_msg_handlers[] = {
+	{
+		.type		= QMI_RESPONSE,
+		.msg_id		= IPA_QMI_INIT_DRIVER,
+		.ei		= ipa_init_modem_driver_rsp_ei,
+		.decoded_size	= IPA_QMI_INIT_DRIVER_RSP_SZ,
+		.fn		= ipa_init_driver_rsp_fn,
+	},
+};
+
+/* Return a pointer to an init modem driver request structure, which contains
+ * configuration parameters for the modem.  The modem may be started multiple
+ * times, but generally these parameters don't change so we can reuse the
+ * request structure once it's initialized.  The only exception is the
+ * skip_uc_load field, which will be set only after the microcontroller has
+ * reported it has completed its initialization.
+ */
+static const struct ipa_init_modem_driver_req *
+init_modem_driver_req(struct ipa_qmi *ipa_qmi)
+{
+	struct ipa *ipa = container_of(ipa_qmi, struct ipa, qmi);
+	static struct ipa_init_modem_driver_req req;
+
+	/* This is not the first boot if the microcontroller is loaded */
+	req.skip_uc_load = ipa->uc_loaded;
+	req.skip_uc_load_valid = true;
+
+	/* We only have to initialize most of it once */
+	if (req.platform_type_valid)
+		return &req;
+
+	req.platform_type_valid = true;
+	req.platform_type = IPA_QMI_PLATFORM_TYPE_MSM_ANDROID;
+
+	req.hdr_tbl_info_valid = IPA_SMEM_MODEM_HDR_SIZE ? 1 : 0;
+	req.hdr_tbl_info.start = ipa_qmi->base + IPA_SMEM_MODEM_HDR_OFFSET;
+	req.hdr_tbl_info.end = req.hdr_tbl_info.start +
+					IPA_SMEM_MODEM_HDR_SIZE - 1;
+
+	req.v4_route_tbl_info_valid = true;
+	req.v4_route_tbl_info.start =
+			ipa_qmi->base + IPA_SMEM_V4_RT_NHASH_OFFSET;
+	req.v4_route_tbl_info.count = IPA_SMEM_MODEM_RT_COUNT;
+
+	req.v6_route_tbl_info_valid = true;
+	req.v6_route_tbl_info.start =
+			ipa_qmi->base + IPA_SMEM_V6_RT_NHASH_OFFSET;
+	req.v6_route_tbl_info.count = IPA_SMEM_MODEM_RT_COUNT;
+
+	req.v4_filter_tbl_start_valid = true;
+	req.v4_filter_tbl_start = ipa_qmi->base + IPA_SMEM_V4_FLT_NHASH_OFFSET;
+
+	req.v6_filter_tbl_start_valid = true;
+	req.v6_filter_tbl_start = ipa_qmi->base + IPA_SMEM_V6_FLT_NHASH_OFFSET;
+
+	req.modem_mem_info_valid = IPA_SMEM_MODEM_SIZE ? 1 : 0;
+	req.modem_mem_info.start = ipa_qmi->base + IPA_SMEM_MODEM_OFFSET;
+	req.modem_mem_info.size = IPA_SMEM_MODEM_SIZE;
+
+	req.ctrl_comm_dest_end_pt_valid = true;
+	req.ctrl_comm_dest_end_pt = IPA_ENDPOINT_AP_MODEM_RX;
+
+	req.hdr_proc_ctx_tbl_info_valid =
+			IPA_SMEM_MODEM_HDR_PROC_CTX_SIZE ? 1 : 0;
+	req.hdr_proc_ctx_tbl_info.start =
+			ipa_qmi->base + IPA_SMEM_MODEM_HDR_PROC_CTX_OFFSET;
+	req.hdr_proc_ctx_tbl_info.end = req.hdr_proc_ctx_tbl_info.start +
+			IPA_SMEM_MODEM_HDR_PROC_CTX_SIZE - 1;
+
+	req.v4_hash_route_tbl_info_valid = true;
+	req.v4_hash_route_tbl_info.start =
+			ipa_qmi->base + IPA_SMEM_V4_RT_HASH_OFFSET;
+	req.v4_hash_route_tbl_info.count = IPA_SMEM_MODEM_RT_COUNT;
+
+	req.v6_hash_route_tbl_info_valid = true;
+	req.v6_hash_route_tbl_info.start =
+			ipa_qmi->base + IPA_SMEM_V6_RT_HASH_OFFSET;
+	req.v6_hash_route_tbl_info.count = IPA_SMEM_MODEM_RT_COUNT;
+
+	req.v4_hash_filter_tbl_start_valid = true;
+	req.v4_hash_filter_tbl_start =
+			ipa_qmi->base + IPA_SMEM_V4_FLT_HASH_OFFSET;
+
+	req.v6_hash_filter_tbl_start_valid = true;
+	req.v6_hash_filter_tbl_start =
+			ipa_qmi->base + IPA_SMEM_V6_FLT_HASH_OFFSET;
+
+	return &req;
+}
+
+/* The modem service we requested is now available via the client handle.
+ * Send an INIT_DRIVER request to the modem.
+ */
+static int
+ipa_client_new_server(struct qmi_handle *qmi, struct qmi_service *svc)
+{
+	const struct ipa_init_modem_driver_req *req;
+	struct ipa *ipa;
+	struct sockaddr_qrtr sq;
+	struct qmi_txn *txn;
+	int ret;
+
+	ipa = container_of(qmi, struct ipa, qmi.client_handle);
+	req = init_modem_driver_req(&ipa->qmi);
+
+	txn = kzalloc(sizeof(*txn), GFP_KERNEL);
+	if (!txn)
+		return -ENOMEM;
+
+	ret = qmi_txn_init(qmi, txn, NULL, NULL);
+	if (ret) {
+		kfree(txn);
+		return ret;
+	}
+
+	sq.sq_family = AF_QIPCRTR;
+	sq.sq_node = svc->node;
+	sq.sq_port = svc->port;
+
+	ret = qmi_send_request(qmi, &sq, txn, IPA_QMI_INIT_DRIVER,
+			       IPA_QMI_INIT_DRIVER_REQ_SZ,
+			       ipa_init_modem_driver_req_ei, req);
+	if (!ret)
+		ret = qmi_txn_wait(txn, MAX_SCHEDULE_TIMEOUT);
+	if (ret)
+		qmi_txn_cancel(txn);
+	kfree(txn);
+
+	return ret;
+}
+
+/* The only callback we supply for the client handle is notification that the
+ * service on the modem has become available.
+ */
+static struct qmi_ops ipa_client_ops = {
+	.new_server	= ipa_client_new_server,
+};
+
+static int ipa_qmi_initialize(struct ipa *ipa)
+{
+	struct ipa_qmi *ipa_qmi = &ipa->qmi;
+	int ret;
+
+	/* The only handle operation that might be interesting for the server
+	 * would be del_client, to find out when the modem side client has
+	 * disappeared.  But other than reporting the event, we wouldn't do
+	 * anything about that.  So we just pass a null pointer for its handle
+	 * operations.  All the real work is done by the message handlers.
+	 */
+	ret = qmi_handle_init(&ipa_qmi->server_handle,
+			      IPA_QMI_SERVER_MAX_RCV_SZ, NULL,
+			      ipa_server_msg_handlers);
+	if (ret)
+		return ret;
+
+	ret = qmi_add_server(&ipa_qmi->server_handle, IPA_HOST_SERVICE_SVC_ID,
+			     IPA_HOST_SVC_VERS, IPA_HOST_SERVICE_INS_ID);
+	if (ret)
+		goto err_release_server_handle;
+
+	/* The client handle is only used for sending an INIT_DRIVER request
+	 * to the modem, and receiving its response message.
+	 */
+	ret = qmi_handle_init(&ipa_qmi->client_handle,
+			      IPA_QMI_CLIENT_MAX_RCV_SZ, &ipa_client_ops,
+			      ipa_client_msg_handlers);
+	if (ret)
+		goto err_release_server_handle;
+
+	ret = qmi_add_lookup(&ipa_qmi->client_handle, IPA_MODEM_SERVICE_SVC_ID,
+			     IPA_MODEM_SVC_VERS, IPA_MODEM_SERVICE_INS_ID);
+	if (ret)
+		goto err_release_client_handle;
+
+	/* All QMI offsets are relative to the start of IPA shared memory */
+	ipa_qmi->base = ipa->shared_offset;
+	ipa_qmi->initialized = 1;
+
+	return 0;
+
+err_release_client_handle:
+	/* Releasing the handle also removes registered lookups */
+	qmi_handle_release(&ipa_qmi->client_handle);
+	memset(&ipa_qmi->client_handle, 0, sizeof(ipa_qmi->client_handle));
+err_release_server_handle:
+	/* Releasing the handle also removes registered services */
+	qmi_handle_release(&ipa_qmi->server_handle);
+	memset(&ipa_qmi->server_handle, 0, sizeof(ipa_qmi->server_handle));
+
+	return ret;
+}
+
+/* This is called by ipa_netdev_setup().  We can be informed via remoteproc
+ * that the modem has shut down, in which case this function will be called
+ * again to prepare for it coming back up again.
+ */
+int ipa_qmi_setup(struct ipa *ipa)
+{
+	ipa->qmi.init_driver_response_received = 0;
+	ipa->qmi.indication_register_received = 0;
+
+	if (!ipa->qmi.initialized)
+		return ipa_qmi_initialize(ipa);
+
+	return 0;
+}
+
+void ipa_qmi_teardown(struct ipa *ipa)
+{
+	if (!ipa->qmi.initialized)
+		return;
+
+	qmi_handle_release(&ipa->qmi.client_handle);
+	memset(&ipa->qmi.client_handle, 0, sizeof(ipa->qmi.client_handle));
+
+	qmi_handle_release(&ipa->qmi.server_handle);
+	memset(&ipa->qmi.server_handle, 0, sizeof(ipa->qmi.server_handle));
+
+	ipa->qmi.initialized = 0;
+}
drivers/net/ipa/ipa_qmi.h new file +35 -0
new file mode 100644
index 000000000000..cfdafa23cf8f
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Copyright (c) 2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018-2019 Linaro Ltd.
+ */
+#ifndef _IPA_QMI_H_
+#define _IPA_QMI_H_
+
+#include <linux/types.h>
+#include <linux/soc/qcom/qmi.h>
+
+struct ipa;
+
+/**
+ * struct ipa_qmi - QMI state associated with an IPA
+ * @initialized		- whether QMI initialization has completed
+ * @client_handle	- used to send an QMI requests to the modem
+ * @server_handle	- used to handle QMI requests from the modem
+ * @indication_register_received - tracks modem request receipt
+ * @init_driver_response_received - tracks modem response receipt
+ */
+struct ipa_qmi {
+	u32 initialized;
+	u32 base;
+	struct qmi_handle client_handle;
+	struct qmi_handle server_handle;
+	u32 indication_register_received;
+	u32 init_driver_response_received;
+
+};
+
+int ipa_qmi_setup(struct ipa *ipa);
+void ipa_qmi_teardown(struct ipa *ipa);
+
+#endif /* !_IPA_QMI_H_ */
drivers/net/ipa/ipa_qmi_msg.c new file +583 -0

583 lines not on this page · raw message

drivers/net/ipa/ipa_qmi_msg.h new file +238 -0

238 lines not on this page · raw message

drivers/net/ipa/ipa_smp2p.c new file +304 -0

304 lines not on this page · raw message

drivers/net/ipa/ipa_smp2p.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#19 lore raw

PATCH v2 14/17 soc: qcom: ipa: support build of IPA code

In reply to Alex Elder

Add build and Kconfig support for the Qualcomm IPA driver.

---
 drivers/net/Kconfig      |  2 ++
 drivers/net/Makefile     |  1 +
 drivers/net/ipa/Kconfig  | 16 ++++++++++++++++
 drivers/net/ipa/Makefile |  7 +++++++
 4 files changed, 26 insertions(+)
 create mode 100644 drivers/net/ipa/Kconfig
 create mode 100644 drivers/net/ipa/Makefile
drivers/net/Kconfig +2 -0
index 48e209e55843..d87fe174eb9f 100644
@@ -388,6 +388,8 @@ source "drivers/net/fddi/Kconfig"
 
 source "drivers/net/hippi/Kconfig"
 
+source "drivers/net/ipa/Kconfig"
+
 config NET_SB1000
 	tristate "General Instruments Surfboard 1000"
 	depends on PNP
drivers/net/Makefile +1 -0
index 0d3ba056cda3..ff8918fe09b0 100644
@@ -45,6 +45,7 @@ obj-$(CONFIG_ETHERNET) += ethernet/
 obj-$(CONFIG_FDDI) += fddi/
 obj-$(CONFIG_HIPPI) += hippi/
 obj-$(CONFIG_HAMRADIO) += hamradio/
+obj-$(CONFIG_IPA) += ipa/
 obj-$(CONFIG_PLIP) += plip/
 obj-$(CONFIG_PPP) += ppp/
 obj-$(CONFIG_PPP_ASYNC) += ppp/
drivers/net/ipa/Kconfig new file +16 -0
new file mode 100644
index 000000000000..b1e3f7405992
@@ -0,0 +1,16 @@
+config IPA
+	tristate "Qualcomm IPA support"
+	depends on NET
+	select QCOM_QMI_HELPERS
+	select QCOM_MDT_LOADER
+	default n
+	help
+	  Choose Y here to include support for the Qualcomm IP Accelerator
+	  (IPA), a hardware block present in some Qualcomm SoCs.  The IPA
+	  is a programmable protocol processor that is capable of generic
+	  hardware handling of IP packets, including routing, filtering,
+	  and NAT.  Currently the IPA driver supports only basic transport
+	  of network traffic between the AP and modem, on the Qualcomm
+	  SDM845 SoC.
+
+	  If unsure, say N.
drivers/net/ipa/Makefile new file +7 -0
new file mode 100644
index 000000000000..a43039c09a25
@@ -0,0 +1,7 @@
+obj-$(CONFIG_IPA)	+=	ipa.o
+
+ipa-y			:=	ipa_main.o ipa_clock.o ipa_mem.o \
+				ipa_interrupt.o gsi.o gsi_trans.o \
+				ipa_gsi.o ipa_smp2p.o ipa_uc.o \
+				ipa_endpoint.o ipa_cmd.o ipa_netdev.o \
+				ipa_qmi.o ipa_qmi_msg.o ipa_data-sdm845.o
Signature · 1 line
2.20.1
author
#20 lore raw

PATCH v2 15/17 MAINTAINERS: add entry for the Qualcomm IPA driver

In reply to Alex Elder

Add an entry in the MAINTAINERS file for the Qualcomm IPA driver

---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)
MAINTAINERS +6 -0
index 429c6c624861..a2dece647641 100644
@@ -12872,6 +12872,12 @@ L:	[email protected] (moderated for non-subscribers)
 S:	Supported
 F:	sound/soc/qcom/
 
+QCOM IPA DRIVER
+M:	Alex Elder <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/net/ipa/
+
 QEMU MACHINE EMULATOR AND VIRTUALIZER SUPPORT
 M:	Gabriel Somlo <[email protected]>
 M:	"Michael S. Tsirkin" <[email protected]>
Signature · 1 line
2.20.1
author
#21 lore raw

PATCH v2 16/17 arm64: dts: sdm845: add IPA information

In reply to Alex Elder

Add IPA-related nodes and definitions to "sdm845.dtsi".

---
 arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 ++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
arch/arm64/boot/dts/qcom/sdm845.dtsi +51 -0
index fcb93300ca62..985479925af8 100644
@@ -20,6 +20,7 @@
 #include <dt-bindings/soc/qcom,rpmh-rsc.h>
 #include <dt-bindings/clock/qcom,gcc-sdm845.h>
 #include <dt-bindings/thermal/thermal.h>
+#include <dt-bindings/interconnect/qcom,sdm845.h>
 
 / {
 	interrupt-parent = <&intc>;
@@ -517,6 +518,17 @@
 			interrupt-controller;
 			#interrupt-cells = <2>;
 		};
+
+		ipa_smp2p_out: ipa-ap-to-modem {
+			qcom,entry-name = "ipa";
+			#qcom,smem-state-cells = <1>;
+		};
+
+		ipa_smp2p_in: ipa-modem-to-ap {
+			qcom,entry-name = "ipa";
+			interrupt-controller;
+			#interrupt-cells = <2>;
+		};
 	};
 
 	smp2p-slpi {
@@ -1268,6 +1280,45 @@
 			};
 		};
 
+		ipa@1e40000 {
+			compatible = "qcom,sdm845-ipa";
+
+			modem-init;
+
+			reg = <0 0x1e40000 0 0x7000>,
+			      <0 0x1e47000 0 0x2000>,
+			      <0 0x1e04000 0 0x2c000>;
+			reg-names = "ipa-reg",
+				    "ipa-shared",
+				    "gsi";
+
+			interrupts-extended =
+					<&intc 0 311 IRQ_TYPE_EDGE_RISING>,
+					<&intc 0 432 IRQ_TYPE_LEVEL_HIGH>,
+					<&ipa_smp2p_in 0 IRQ_TYPE_EDGE_RISING>,
+					<&ipa_smp2p_in 1 IRQ_TYPE_EDGE_RISING>;
+			interrupt-names = "ipa",
+					  "gsi",
+					  "ipa-clock-query",
+					  "ipa-setup-ready";
+
+			clocks = <&rpmhcc RPMH_IPA_CLK>;
+			clock-names = "core";
+
+			interconnects =
+				<&rsc_hlos MASTER_IPA &rsc_hlos SLAVE_EBI1>,
+				<&rsc_hlos MASTER_IPA &rsc_hlos SLAVE_IMEM>,
+				<&rsc_hlos MASTER_APPSS_PROC &rsc_hlos SLAVE_IPA_CFG>;
+			interconnect-names = "memory",
+					     "imem",
+					     "config";
+
+			qcom,smem-states = <&ipa_smp2p_out 0>,
+					   <&ipa_smp2p_out 1>;
+			qcom,smem-state-names = "ipa-clock-enabled-valid",
+						"ipa-clock-enabled";
+		};
+
 		tcsr_mutex_regs: syscon@1f40000 {
 			compatible = "syscon";
 			reg = <0 0x01f40000 0 0x40000>;
Signature · 1 line
2.20.1
author
#22 lore raw

PATCH v2 17/17 arm64: defconfig: enable build of IPA code

In reply to Alex Elder

Add CONFIG_IPA to the 64-bit Arm defconfig.

---
 arch/arm64/configs/defconfig | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/configs/defconfig +1 -0
index 4d583514258c..6ed86cb6b597 100644
@@ -261,6 +261,7 @@ CONFIG_SMSC911X=y
 CONFIG_SNI_AVE=y
 CONFIG_SNI_NETSEC=y
 CONFIG_STMMAC_ETH=m
+CONFIG_IPA=m
 CONFIG_MDIO_BUS_MUX_MMIOREG=y
 CONFIG_AT803X_PHY=m
 CONFIG_MARVELL_PHY=m
Signature · 1 line
2.20.1
#23 lore raw
In reply to Alex Elder
Quoted 35 lines · Alex Elder

This series presents the driver for the Qualcomm IP Accelerator (IPA).

This is version 2 of the series. This version has addressed almost all of the feedback received in the first version:

https://lore.kernel.org/lkml/[email protected]/
More detail is included in the individual patches, but here is a high-level summary of what's changed since then:
- Two spinlocks have been removed.
- The code for enabling and disabling endpoint interrupts has been simplified considerably, and the spinlock is no longer required
- A spinlock used when updating ring buffer pointers is no longer needed. Integers indexing the ring are used instead (and they don't even have to be atomic).
- One spinlock remains to protect list updates, but it is always acquired using spin_lock_bh() (no more irqsave).
- Information about the queueing and completion of messages is now supplied to the network stack in batches rather than one at a time.
- I/O completion handling has been simplified, with the IRQ handler now consisting mainly of disabling the interrupt and calling napi_schedule().
- Some comments have been updated and improved througout.

What follows is the introduction supplied with v1 of the series.

-----

The IPA is a component present in some Qualcomm SoCs that allows network functions such as aggregation, filtering, routing, and NAT to be performed without active involvement of the main application processor (AP).

Quoted 4 lines

Initially, these advanced features are disabled; the IPA driver simply provides a network interface that makes the modem's LTE network available to the AP. In addition, only support for the IPA found in the Qualcomm SDM845 SoC is provided.

My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC.

Is the firmware that the driver loads already in linux-firmware or going to be there soon?

How does the driver support multiple PDNs (eg PDP or EPS contexts) that are enabled through the control plane via QMI messages? I couldn't quite find that out.

Thanks,
Dan

Quoted 139 lines

This code is derived from a driver developed internally by Qualcomm. A version of the original source can be seen here:
https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree
in the "drivers/platform/msm/ipa" directory. Many were involved in developing this, but the following individuals deserve explicit acknowledgement for their substantial contributions:

Abhishek Choubey
Ady Abraham
Chaitanya Pratapa
David Arinzon
Ghanim Fodi
Gidon Studinski
Ravi Gummadidala
Shihuan Liu
Skylar Chang

A version of this code was posted in November 2018 as an RFC.

https://lore.kernel.org/lkml/[email protected]/
All feedback received was addressed. The code has undergone considerable further rework since that time, and most of the "future work" described then has now been completed.

This code is available in buildable form here, based on kernel v5.2-rc1:
remote: ssh://[email protected]/people/alex.elder/linux.git
branch: ipa-v2_kernel-v5.2-rc2
75adf2ac1266 arm64: defconfig: enable build of IPA code

The branch depends on a commit now found in in net-next. It has been cherry-picked, and (in this branch) has this commit ID:
13c627b5a078 net: qualcomm: rmnet: Move common struct definitions
to include
by

-Alex

Alex Elder (17):
bitfield.h: add FIELD_MAX() and field_max()
dt-bindings: soc: qcom: add IPA bindings
soc: qcom: ipa: main code
soc: qcom: ipa: configuration data
soc: qcom: ipa: clocking, interrupts, and memory
soc: qcom: ipa: GSI headers
soc: qcom: ipa: the generic software interface
soc: qcom: ipa: GSI transactions
soc: qcom: ipa: IPA interface to GSI
soc: qcom: ipa: IPA endpoints
soc: qcom: ipa: immediate commands
soc: qcom: ipa: IPA network device and microcontroller
soc: qcom: ipa: AP/modem communications
soc: qcom: ipa: support build of IPA code
MAINTAINERS: add entry for the Qualcomm IPA driver
arm64: dts: sdm845: add IPA information
arm64: defconfig: enable build of IPA code

 .../devicetree/bindings/net/qcom,ipa.yaml     |  180 ++
 MAINTAINERS                                   |    6 +
 arch/arm64/boot/dts/qcom/sdm845.dtsi          |   51 +
 arch/arm64/configs/defconfig                  |    1 +
 drivers/net/Kconfig                           |    2 +
 drivers/net/Makefile                          |    1 +
 drivers/net/ipa/Kconfig                       |   16 +
 drivers/net/ipa/Makefile                      |    7 +
 drivers/net/ipa/gsi.c                         | 1635
+++++++++++++++++
 drivers/net/ipa/gsi.h                         |  246 +++
 drivers/net/ipa/gsi_private.h                 |  148 ++
 drivers/net/ipa/gsi_reg.h                     |  376 ++++
 drivers/net/ipa/gsi_trans.c                   |  624 +++++++
 drivers/net/ipa/gsi_trans.h                   |  116 ++
 drivers/net/ipa/ipa.h                         |  131 ++
 drivers/net/ipa/ipa_clock.c                   |  297 +++
 drivers/net/ipa/ipa_clock.h                   |   52 +
 drivers/net/ipa/ipa_cmd.c                     |  377 ++++
 drivers/net/ipa/ipa_cmd.h                     |  116 ++
 drivers/net/ipa/ipa_data-sdm845.c             |  245 +++
 drivers/net/ipa/ipa_data.h                    |  267 +++
 drivers/net/ipa/ipa_endpoint.c                | 1283 +++++++++++++
 drivers/net/ipa/ipa_endpoint.h                |   97 +
 drivers/net/ipa/ipa_gsi.c                     |   48 +
 drivers/net/ipa/ipa_gsi.h                     |   49 +
 drivers/net/ipa/ipa_interrupt.c               |  279 +++
 drivers/net/ipa/ipa_interrupt.h               |   53 +
 drivers/net/ipa/ipa_main.c                    |  921 ++++++++++
 drivers/net/ipa/ipa_mem.c                     |  234 +++
 drivers/net/ipa/ipa_mem.h                     |   83 +
 drivers/net/ipa/ipa_netdev.c                  |  251 +++
 drivers/net/ipa/ipa_netdev.h                  |   24 +
 drivers/net/ipa/ipa_qmi.c                     |  402 ++++
 drivers/net/ipa/ipa_qmi.h                     |   35 +
 drivers/net/ipa/ipa_qmi_msg.c                 |  583 ++++++
 drivers/net/ipa/ipa_qmi_msg.h                 |  238 +++
 drivers/net/ipa/ipa_reg.h                     |  279 +++
 drivers/net/ipa/ipa_smp2p.c                   |  304 +++
 drivers/net/ipa/ipa_smp2p.h                   |   47 +
 drivers/net/ipa/ipa_uc.c                      |  208 +++
 drivers/net/ipa/ipa_uc.h                      |   32 +
 include/linux/bitfield.h                      |   14 +
 42 files changed, 10358 insertions(+)
 create mode 100644
Documentation/devicetree/bindings/net/qcom,ipa.yaml
 create mode 100644 drivers/net/ipa/Kconfig
 create mode 100644 drivers/net/ipa/Makefile
 create mode 100644 drivers/net/ipa/gsi.c
 create mode 100644 drivers/net/ipa/gsi.h
 create mode 100644 drivers/net/ipa/gsi_private.h
 create mode 100644 drivers/net/ipa/gsi_reg.h
 create mode 100644 drivers/net/ipa/gsi_trans.c
 create mode 100644 drivers/net/ipa/gsi_trans.h
 create mode 100644 drivers/net/ipa/ipa.h
 create mode 100644 drivers/net/ipa/ipa_clock.c
 create mode 100644 drivers/net/ipa/ipa_clock.h
 create mode 100644 drivers/net/ipa/ipa_cmd.c
 create mode 100644 drivers/net/ipa/ipa_cmd.h
 create mode 100644 drivers/net/ipa/ipa_data-sdm845.c
 create mode 100644 drivers/net/ipa/ipa_data.h
 create mode 100644 drivers/net/ipa/ipa_endpoint.c
 create mode 100644 drivers/net/ipa/ipa_endpoint.h
 create mode 100644 drivers/net/ipa/ipa_gsi.c
 create mode 100644 drivers/net/ipa/ipa_gsi.h
 create mode 100644 drivers/net/ipa/ipa_interrupt.c
 create mode 100644 drivers/net/ipa/ipa_interrupt.h
 create mode 100644 drivers/net/ipa/ipa_main.c
 create mode 100644 drivers/net/ipa/ipa_mem.c
 create mode 100644 drivers/net/ipa/ipa_mem.h
 create mode 100644 drivers/net/ipa/ipa_netdev.c
 create mode 100644 drivers/net/ipa/ipa_netdev.h
 create mode 100644 drivers/net/ipa/ipa_qmi.c
 create mode 100644 drivers/net/ipa/ipa_qmi.h
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.c
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.h
 create mode 100644 drivers/net/ipa/ipa_reg.h
 create mode 100644 drivers/net/ipa/ipa_smp2p.c
 create mode 100644 drivers/net/ipa/ipa_smp2p.h
 create mode 100644 drivers/net/ipa/ipa_uc.c
 create mode 100644 drivers/net/ipa/ipa_uc.h
author
#24 lore raw
Quoted 41 lines · Dan Williams
Quoted 35 lines · Alex Elder

This series presents the driver for the Qualcomm IP Accelerator (IPA).

This is version 2 of the series. This version has addressed almost all of the feedback received in the first version:

https://lore.kernel.org/lkml/[email protected]/
More detail is included in the individual patches, but here is a high-level summary of what's changed since then:
- Two spinlocks have been removed.
- The code for enabling and disabling endpoint interrupts has been simplified considerably, and the spinlock is no longer required
- A spinlock used when updating ring buffer pointers is no longer needed. Integers indexing the ring are used instead (and they don't even have to be atomic).
- One spinlock remains to protect list updates, but it is always acquired using spin_lock_bh() (no more irqsave).
- Information about the queueing and completion of messages is now supplied to the network stack in batches rather than one at a time.
- I/O completion handling has been simplified, with the IRQ handler now consisting mainly of disabling the interrupt and calling napi_schedule().
- Some comments have been updated and improved througout.

What follows is the introduction supplied with v1 of the series.

-----

The IPA is a component present in some Qualcomm SoCs that allows network functions such as aggregation, filtering, routing, and NAT to be performed without active involvement of the main application processor (AP).

Quoted 4 lines

Initially, these advanced features are disabled; the IPA driver simply provides a network interface that makes the modem's LTE network available to the AP. In addition, only support for the IPA found in the Qualcomm SDM845 SoC is provided.

Quoted 6 lines

My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC.

First, the relationship between the IPA driver and the rmnet driver is that the IPA driver is assumed to sit between the rmnet driver and the hardware.

Currently the modem is assumed to use QMAP protocol. This means each packet is prefixed by a (struct rmnet_map_header) structure that allows the IPA connection to be multiplexed for several logical connections. The rmnet driver parses such messages and implements the multiplexed network interfaces.

QMAP protocol can also be used for aggregating many small packets into a larger message. The rmnet driver implements de-aggregation of such messages (and could probably aggregate them for TX as well).

Finally, the IPA can support checksum offload, and the rmnet driver handles providing a prepended header (for TX) and interpreting the appended trailer (for RX) if these features are enabled.

So basically, the purpose of the rmnet driver is to handle QMAP protocol connections, and right now that's what the modem provides.

Quoted 2 lines

Is the firmware that the driver loads already in linux-firmware or going to be there soon?

It is not right now, and I have no information on when it can be available. The AP *can* load the firmware but right now we rely on the modem doing it (until we can make the firmware available).

Quoted 3 lines

How does the driver support multiple PDNs (eg PDP or EPS contexts) that are enabled through the control plane via QMI messages? I couldn't quite find that out.

To be honest, I don't know the answer to this. All of my work has been on this transport driver and I believe these things are handled by user space. But I really don't know details.

-Alex

Quoted 143 lines

Thanks,
Dan

Quoted 139 lines

This code is derived from a driver developed internally by Qualcomm. A version of the original source can be seen here:
https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree
in the "drivers/platform/msm/ipa" directory. Many were involved in developing this, but the following individuals deserve explicit acknowledgement for their substantial contributions:

Abhishek Choubey
Ady Abraham
Chaitanya Pratapa
David Arinzon
Ghanim Fodi
Gidon Studinski
Ravi Gummadidala
Shihuan Liu
Skylar Chang

A version of this code was posted in November 2018 as an RFC.

https://lore.kernel.org/lkml/[email protected]/
All feedback received was addressed. The code has undergone considerable further rework since that time, and most of the "future work" described then has now been completed.

This code is available in buildable form here, based on kernel v5.2-rc1:
remote: ssh://[email protected]/people/alex.elder/linux.git
branch: ipa-v2_kernel-v5.2-rc2
75adf2ac1266 arm64: defconfig: enable build of IPA code

The branch depends on a commit now found in in net-next. It has been cherry-picked, and (in this branch) has this commit ID:
13c627b5a078 net: qualcomm: rmnet: Move common struct definitions
to include
by

-Alex

Alex Elder (17):
bitfield.h: add FIELD_MAX() and field_max()
dt-bindings: soc: qcom: add IPA bindings
soc: qcom: ipa: main code
soc: qcom: ipa: configuration data
soc: qcom: ipa: clocking, interrupts, and memory
soc: qcom: ipa: GSI headers
soc: qcom: ipa: the generic software interface
soc: qcom: ipa: GSI transactions
soc: qcom: ipa: IPA interface to GSI
soc: qcom: ipa: IPA endpoints
soc: qcom: ipa: immediate commands
soc: qcom: ipa: IPA network device and microcontroller
soc: qcom: ipa: AP/modem communications
soc: qcom: ipa: support build of IPA code
MAINTAINERS: add entry for the Qualcomm IPA driver
arm64: dts: sdm845: add IPA information
arm64: defconfig: enable build of IPA code

 .../devicetree/bindings/net/qcom,ipa.yaml     |  180 ++
 MAINTAINERS                                   |    6 +
 arch/arm64/boot/dts/qcom/sdm845.dtsi          |   51 +
 arch/arm64/configs/defconfig                  |    1 +
 drivers/net/Kconfig                           |    2 +
 drivers/net/Makefile                          |    1 +
 drivers/net/ipa/Kconfig                       |   16 +
 drivers/net/ipa/Makefile                      |    7 +
 drivers/net/ipa/gsi.c                         | 1635
+++++++++++++++++
 drivers/net/ipa/gsi.h                         |  246 +++
 drivers/net/ipa/gsi_private.h                 |  148 ++
 drivers/net/ipa/gsi_reg.h                     |  376 ++++
 drivers/net/ipa/gsi_trans.c                   |  624 +++++++
 drivers/net/ipa/gsi_trans.h                   |  116 ++
 drivers/net/ipa/ipa.h                         |  131 ++
 drivers/net/ipa/ipa_clock.c                   |  297 +++
 drivers/net/ipa/ipa_clock.h                   |   52 +
 drivers/net/ipa/ipa_cmd.c                     |  377 ++++
 drivers/net/ipa/ipa_cmd.h                     |  116 ++
 drivers/net/ipa/ipa_data-sdm845.c             |  245 +++
 drivers/net/ipa/ipa_data.h                    |  267 +++
 drivers/net/ipa/ipa_endpoint.c                | 1283 +++++++++++++
 drivers/net/ipa/ipa_endpoint.h                |   97 +
 drivers/net/ipa/ipa_gsi.c                     |   48 +
 drivers/net/ipa/ipa_gsi.h                     |   49 +
 drivers/net/ipa/ipa_interrupt.c               |  279 +++
 drivers/net/ipa/ipa_interrupt.h               |   53 +
 drivers/net/ipa/ipa_main.c                    |  921 ++++++++++
 drivers/net/ipa/ipa_mem.c                     |  234 +++
 drivers/net/ipa/ipa_mem.h                     |   83 +
 drivers/net/ipa/ipa_netdev.c                  |  251 +++
 drivers/net/ipa/ipa_netdev.h                  |   24 +
 drivers/net/ipa/ipa_qmi.c                     |  402 ++++
 drivers/net/ipa/ipa_qmi.h                     |   35 +
 drivers/net/ipa/ipa_qmi_msg.c                 |  583 ++++++
 drivers/net/ipa/ipa_qmi_msg.h                 |  238 +++
 drivers/net/ipa/ipa_reg.h                     |  279 +++
 drivers/net/ipa/ipa_smp2p.c                   |  304 +++
 drivers/net/ipa/ipa_smp2p.h                   |   47 +
 drivers/net/ipa/ipa_uc.c                      |  208 +++
 drivers/net/ipa/ipa_uc.h                      |   32 +
 include/linux/bitfield.h                      |   14 +
 42 files changed, 10358 insertions(+)
 create mode 100644
Documentation/devicetree/bindings/net/qcom,ipa.yaml
 create mode 100644 drivers/net/ipa/Kconfig
 create mode 100644 drivers/net/ipa/Makefile
 create mode 100644 drivers/net/ipa/gsi.c
 create mode 100644 drivers/net/ipa/gsi.h
 create mode 100644 drivers/net/ipa/gsi_private.h
 create mode 100644 drivers/net/ipa/gsi_reg.h
 create mode 100644 drivers/net/ipa/gsi_trans.c
 create mode 100644 drivers/net/ipa/gsi_trans.h
 create mode 100644 drivers/net/ipa/ipa.h
 create mode 100644 drivers/net/ipa/ipa_clock.c
 create mode 100644 drivers/net/ipa/ipa_clock.h
 create mode 100644 drivers/net/ipa/ipa_cmd.c
 create mode 100644 drivers/net/ipa/ipa_cmd.h
 create mode 100644 drivers/net/ipa/ipa_data-sdm845.c
 create mode 100644 drivers/net/ipa/ipa_data.h
 create mode 100644 drivers/net/ipa/ipa_endpoint.c
 create mode 100644 drivers/net/ipa/ipa_endpoint.h
 create mode 100644 drivers/net/ipa/ipa_gsi.c
 create mode 100644 drivers/net/ipa/ipa_gsi.h
 create mode 100644 drivers/net/ipa/ipa_interrupt.c
 create mode 100644 drivers/net/ipa/ipa_interrupt.h
 create mode 100644 drivers/net/ipa/ipa_main.c
 create mode 100644 drivers/net/ipa/ipa_mem.c
 create mode 100644 drivers/net/ipa/ipa_mem.h
 create mode 100644 drivers/net/ipa/ipa_netdev.c
 create mode 100644 drivers/net/ipa/ipa_netdev.h
 create mode 100644 drivers/net/ipa/ipa_qmi.c
 create mode 100644 drivers/net/ipa/ipa_qmi.h
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.c
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.h
 create mode 100644 drivers/net/ipa/ipa_reg.h
 create mode 100644 drivers/net/ipa/ipa_smp2p.c
 create mode 100644 drivers/net/ipa/ipa_smp2p.h
 create mode 100644 drivers/net/ipa/ipa_uc.c
 create mode 100644 drivers/net/ipa/ipa_uc.h
#25 lore raw
Quoted 10 lines · Alex Elder
Quoted 8 lines · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC.

Quoted 3 lines

First, the relationship between the IPA driver and the rmnet driver is that the IPA driver is assumed to sit between the rmnet driver and the hardware.

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

Quoted 10 lines

Currently the modem is assumed to use QMAP protocol. This means each packet is prefixed by a (struct rmnet_map_header) structure that allows the IPA connection to be multiplexed for several logical connections. The rmnet driver parses such messages and implements the multiplexed network interfaces.

QMAP protocol can also be used for aggregating many small packets into a larger message. The rmnet driver implements de-aggregation of such messages (and could probably aggregate them for TX as well).

Quoted 7 lines

Finally, the IPA can support checksum offload, and the rmnet driver handles providing a prepended header (for TX) and interpreting the appended trailer (for RX) if these features are enabled.

So basically, the purpose of the rmnet driver is to handle QMAP protocol connections, and right now that's what the modem provides.

Do you have any idea why this particular design was picked?

My best guess is that it evolved organically with multiple generations of hardware and software, rather than being thought out as a nice abstraction layer. If the two are tightly connected, this might mean that what we actually want here is to reintegrate the two components into a single driver with a much simpler RX and TX path that handles the checksumming and aggregation of data packets directly as it passes them from the network stack into the hardware.

Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween. The intermediate buffer here acts like a router that must pass data along or randomly drop packets when the consumer can't keep up with the producer.

Arnd

author
#26 lore raw
Quoted 11 lines · Arnd Bergmann
Quoted 10 lines · Alex Elder
Quoted 8 lines · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC.

Quoted 7 lines
Quoted 3 lines

First, the relationship between the IPA driver and the rmnet driver is that the IPA driver is assumed to sit between the rmnet driver and the hardware.

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

Quoted 15 lines
Quoted 15 lines

Currently the modem is assumed to use QMAP protocol. This means each packet is prefixed by a (struct rmnet_map_header) structure that allows the IPA connection to be multiplexed for several logical connections. The rmnet driver parses such messages and implements the multiplexed network interfaces.

QMAP protocol can also be used for aggregating many small packets into a larger message. The rmnet driver implements de-aggregation of such messages (and could probably aggregate them for TX as well).

Finally, the IPA can support checksum offload, and the rmnet driver handles providing a prepended header (for TX) and interpreting the appended trailer (for RX) if these features are enabled.

Quoted 4 lines
Quoted 2 lines

So basically, the purpose of the rmnet driver is to handle QMAP protocol connections, and right now that's what the modem provides.

Do you have any idea why this particular design was picked?

I don't really. I inherited it. Early on, when I asked about the need for QMAP I was told it was important because it offered certain features, but at that time I was somewhat new to the code and didn't have the insight to judge the merits of the design. Since then I've mostly just accepted it and concentrated on improving the IPA driver.

Quoted 8 lines

My best guess is that it evolved organically with multiple generations of hardware and software, rather than being thought out as a nice abstraction layer. If the two are tightly connected, this might mean that what we actually want here is to reintegrate the two components into a single driver with a much simpler RX and TX path that handles the checksumming and aggregation of data packets directly as it passes them from the network stack into the hardware.

In general, I agree. And Dan suggested combining the rmnet and IPA drivers into a single driver when I posted the RFC code last year. There's still the notion of switching back ends that I mentioned earlier; if that's indeed an important feature it might argue for keeping rmnet as a shim layer. But I'm really not the person to comment on this. Someone (Subash?) from Qualcomm might be able to provide better answers.

Quoted 3 lines

Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween.

My impression is the rmnet driver is a pretty thin layer, so the CPU overhead is probably not that great (though deaggregating a message is expensive). I agree with you on the flow control.

Quoted 3 lines

The intermediate buffer here acts like a router that must pass data along or randomly drop packets when the consumer can't keep up with the producer.

I haven't reviewed the rmnet code in any detail, but you may be right.

-Alex

Quoted 3 lines

Arnd

#27 lore raw
Quoted 22 lines · Alex Elder
Quoted 7 lines · Arnd Bergmann
Quoted 2 lines · Alex Elder
Quoted 1 line · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

Quoted 9 lines

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

Why would that require firmware changes? What I was thinking here is to turn the bits of the rmnet driver that actually do anything interesting on the headers into a library module (or a header file with inline functions) that can be called directly by the ipa driver, keeping the protocol unchanged.

Quoted 8 lines
Quoted 3 lines

Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween.

My impression is the rmnet driver is a pretty thin layer, so the CPU overhead is probably not that great (though deaggregating a message is expensive). I agree with you on the flow control.

The CPU overhead I mean is not from executing code in the rmnet driver, but from passing packets through the network stack between the two drivers, i.e. adding each frame to a queue and taking it back out. I'm not sure how this ends up working in reality but from a first look it seems like we might bounce in an out of the softirq handler inbetween.

Arnd

maintainer
#7 lore raw

Re: [PATCH v2 03/17] soc: qcom: ipa: main code

In reply to Alex Elder

From: Alex Elder <[email protected]>
Date: Thu, 30 May 2019 22:53:34 -0500

Quoted 1 line
drivers/net/ipa/ipa.h new file +1 -0
+	void *route_virt;

...

drivers/net/ipa/ipa.h line 70

Quoted 1 line
drivers/net/ipa/ipa.h new file +1 -0
+	void *filter_virt;

...

If these are arrays of u64's, please declare them as "u64 *" instead of the opaque "void *".

author
#28 lore raw
In reply to Arnd Bergmann
Quoted 33 lines · Arnd Bergmann
Quoted 22 lines · Alex Elder
Quoted 7 lines · Arnd Bergmann
Quoted 2 lines · Alex Elder
Quoted 1 line · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

Quoted 9 lines

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

Quoted 5 lines

Why would that require firmware changes? What I was thinking here is to turn the bits of the rmnet driver that actually do anything interesting on the headers into a library module (or a header file with inline functions) that can be called directly by the ipa driver, keeping the protocol unchanged.

You know, it's possible you're right about not needing firmware changes. But it has always been my impression they would be needed. Here's why.

It looks like this:

GSI Channel GSI Channel
| |

  ----------   v   -------   v   -------------
  | AP (ep)|=======| IPA |=======|(ep) Modem |
  ----------       -------       -------------

The AP and Modem each have IPA endpoints (ep), which use GSI channels, to communicate with the IPA. Each endpoint has configuration options (such as checksum offload). I *thought* that the configurations of the two endpoints need to be compatible (e.g., they need to agree on whether they're aggregating). But with your questioning I now think you may be right, that only the local endpoint's configuration matters.

I will inquire further on this. I *know* that the AP and modem exchange some information about IPA configuration, but looking more closely that looks like it's all about the configuration of shared IPA resources, not endpoints.

That said, the broader design (including the user space code) surely assumes rmnet, and I don't have any sense of what impact changing that would make. I am sure that changing it would not be well received.

-Alex

Quoted 18 lines
Quoted 8 lines
Quoted 3 lines

Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween.

My impression is the rmnet driver is a pretty thin layer, so the CPU overhead is probably not that great (though deaggregating a message is expensive). I agree with you on the flow control.

The CPU overhead I mean is not from executing code in the rmnet driver, but from passing packets through the network stack between the two drivers, i.e. adding each frame to a queue and taking it back out. I'm not sure how this ends up working in reality but from a first look it seems like we might bounce in an out of the softirq handler inbetween.

Arnd

author
#8 lore raw

Re: [PATCH v2 03/17] soc: qcom: ipa: main code

In reply to David Miller
Quoted 10 lines · David Miller

From: Alex Elder <[email protected]>
Date: Thu, 30 May 2019 22:53:34 -0500

Quoted 1 line
drivers/net/ipa/ipa.h new file +1 -0
+	void *route_virt;

...

drivers/net/ipa/ipa.h line 70

Quoted 1 line
drivers/net/ipa/ipa.h new file +1 -0
+	void *filter_virt;

...

If these are arrays of u64's, please declare them as "u64 *" instead of the opaque "void *".

Good idea. I hadn't paid attention to that. These tables are arrays of 64-bit addresses so it's better to represent them that way.

Thanks.

-Alex

#96 lore raw
In reply to Arnd Bergmann
Quoted 3 lines · Arnd Bergmann
Quoted 2 lines · Alex Elder
Quoted 1 line · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

[..]

Quoted 7 lines
Quoted 2 lines

So basically, the purpose of the rmnet driver is to handle QMAP protocol connections, and right now that's what the modem provides.

Do you have any idea why this particular design was picked?

From what I've seen of QMAP it seems like a reasonable design choice to

have a software component (rmnet) dealing with this, separate from the transport. And I think IPA is the 4th or 5th mechanism for transporting QMAP packets back and forth to the modem.

Downstream rmnet is copyright 2007-, and I know of interest in bringing at least one of the other transports upstream.

Regards,
Bjorn

#30 lore raw
In reply to Alex Elder
Quoted 33 lines · Alex Elder
Quoted 11 lines · Arnd Bergmann
Quoted 10 lines · Alex Elder
Quoted 8 lines · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC.

Quoted 7 lines
Quoted 3 lines

First, the relationship between the IPA driver and the rmnet driver is that the IPA driver is assumed to sit between the rmnet driver and the hardware.

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

Quoted 10 lines

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

But any such changes would either be years into the future or for specific devices and as such not applicable to any/most of devices on the market now or in the coming years.

But as Arnd points out, if the software split between IPA and rmnet is suboptimal your are encouraged to fix that.

Regards,
Bjorn

#31 lore raw
Quoted 51 lines · Bjorn Andersson
Quoted 33 lines · Alex Elder
Quoted 11 lines · Arnd Bergmann
Quoted 10 lines · Alex Elder
Quoted 8 lines · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC.

Quoted 7 lines
Quoted 3 lines

First, the relationship between the IPA driver and the rmnet driver is that the IPA driver is assumed to sit between the rmnet driver and the hardware.

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

Quoted 10 lines

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

But any such changes would either be years into the future or for specific devices and as such not applicable to any/most of devices on the market now or in the coming years.

Quoted 5 lines

But as Arnd points out, if the software split between IPA and rmnet is suboptimal your are encouraged to fix that.

Regards,
Bjorn

The split rmnet design was chosen because we could place rmnet over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159)
or USB.

rmnet registers a rx handler, so the rmnet packet processing itself happens in the same softirq when packets are queued to network stack by IPA.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#32 lore raw
Quoted 11 lines · Subash Abhinov Kasiviswanathan
Quoted 9 lines · Bjorn Andersson
Quoted 1 line · Alex Elder

On 5/31/19 2:19 PM, Arnd Bergmann wrote:

But any such changes would either be years into the future or for specific devices and as such not applicable to any/most of devices on the market now or in the coming years.

But as Arnd points out, if the software split between IPA and rmnet is suboptimal your are encouraged to fix that.

Quoted 7 lines

The split rmnet design was chosen because we could place rmnet over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159)
or USB.

rmnet registers a rx handler, so the rmnet packet processing itself happens in the same softirq when packets are queued to network stack by IPA.

I've read up on the implementation some more, and concluded that it's mostly a regular protocol wrapper, doing IP over QMAP. There is nothing wrong with the basic concept I think, and as you describe this is an abstraction to keep the common bits in one place, and have them configured consistently.

A few observations on more details here:

- What I'm worried about most here is the flow control handling on the transmit side. The IPA driver now uses the modern BQL method to control how much data gets submitted to the hardware at any time. The rmnet driver also uses flow control using the rmnet_map_command() function, that blocks tx on the higher level device when the remote side asks us to.
I fear that doing flow control for a single physical device on two separate netdev instances is counterproductive and confuses both sides.

- I was a little confused by the location of the rmnet driver in drivers/net/ethernet/... More conventionally, I think as a protocol handler it should go into net/qmap/, with the ipa driver going into drivers/net/qmap/ipa/, similar to what we have fo ethernet, wireless, ppp, appletalk, etc.

- The rx_handler uses gro_cells, which as I understand is meant for generic tunnelling setups and takes another loop through NAPI to aggregate data from multiple queues, but in case of IPA's single-queue receive calling gro directly would be simpler and more efficient.

- I'm still not sure I understand the purpose of the layering with using an rx_handler as opposed to just using EXPORT_SYMBOL(rmnet_rx_handler) and calling that from the hardware driver directly.
From the overall design and the rmnet Kconfig description, it appears as though the intention as that rmnet could be a generic wrapper on top of any device, but from the implementation it seems that IPA is not actually usable that way and would always go through IPA.

Arnd

author
#33 lore raw
Quoted 29 lines · Arnd Bergmann
Quoted 11 lines · Subash Abhinov Kasiviswanathan
Quoted 9 lines · Bjorn Andersson
Quoted 1 line · Alex Elder

On 5/31/19 2:19 PM, Arnd Bergmann wrote:

But any such changes would either be years into the future or for specific devices and as such not applicable to any/most of devices on the market now or in the coming years.

But as Arnd points out, if the software split between IPA and rmnet is suboptimal your are encouraged to fix that.

Quoted 7 lines

The split rmnet design was chosen because we could place rmnet over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159)
or USB.

rmnet registers a rx handler, so the rmnet packet processing itself happens in the same softirq when packets are queued to network stack by IPA.

I've read up on the implementation some more, and concluded that it's mostly a regular protocol wrapper, doing IP over QMAP. There is nothing wrong with the basic concept I think, and as you describe this is an abstraction to keep the common bits in one place, and have them configured consistently.

A few observations on more details here:

Quoted 9 lines

- What I'm worried about most here is the flow control handling on the transmit side. The IPA driver now uses the modern BQL method to control how much data gets submitted to the hardware at any time. The rmnet driver also uses flow control using the rmnet_map_command() function, that blocks tx on the higher level device when the remote side asks us to.
I fear that doing flow control for a single physical device on two separate netdev instances is counterproductive and confuses both sides.

I understand what you're saying here, and instinctively I think you're right.

But BQL manages the *local* interface's ability to get rid of packets, whereas the QMAP flow control is initiated by the other end of the connection (the modem in this case).

With multiplexing, it's possible that one of several logical devices on the modem side has exhausted a resource and must ask the source of the data on the host side to suspend the flow. Meanwhile the other logical devices sharing the physical link might be fine, and should not be delayed by the first one.

It is the multiplexing itself that confuses the BQL algorithm. The abstraction obscures the *real* rates at which individual logical connections are able to transmit data.

Even if the multiple logical interfaces implemented BQL, they would not get the feedback they need directly from the IPA driver, because transmitting over the physical interface might succeed even if the logical interface on the modem side can't handle more data. So I think the flow control commands may be necessary, given multiplexing.

The rmnet driver could use BQL, and could return NETDEV_TX_BUSY for a logical interface when its TX flow has been stopped by a QMAP command. That way the feedback for BQL on the logical interfaces would be provided in the right place.

I have no good intuition about the interaction between two layered BQL managed queues though.

Quoted 11 lines

- I was a little confused by the location of the rmnet driver in drivers/net/ethernet/... More conventionally, I think as a protocol handler it should go into net/qmap/, with the ipa driver going into drivers/net/qmap/ipa/, similar to what we have fo ethernet, wireless, ppp, appletalk, etc.

- The rx_handler uses gro_cells, which as I understand is meant for generic tunnelling setups and takes another loop through NAPI to aggregate data from multiple queues, but in case of IPA's single-queue receive calling gro directly would be simpler and more efficient.

I have been planning to investigate some of the generic GRO stuff for IPA but was going to wait on that until the basic code was upstream.

Quoted 4 lines

- I'm still not sure I understand the purpose of the layering with using an rx_handler as opposed to just using EXPORT_SYMBOL(rmnet_rx_handler) and calling that from the hardware driver directly.

I think that's a good idea.

Quoted 5 lines

From the overall design and the rmnet Kconfig description, it appears as though the intention as that rmnet could be a generic wrapper on top of any device, but from the implementation it seems that IPA is not actually usable that way and would always go through IPA.

As far as I know *nothing* upstream currently uses rmnet; the IPA driver will be the first, but as Bjorn said others seem to be on the way. I'm not sure what you mean by "IPA is not usable that way." Currently the IPA driver assumes a fixed configuration, and that configuration assumes the use of QMAP, and therefore assumes the rmnet driver is layered above it. That doesn't preclude rmnet from using a different back end.

And I'll also mention that although QMAP *can* do multiplexed connections over a single physical link, the IPA code I posted currently supports only one logical interface.

-Alex

Quoted 3 lines

Arnd

#90 lore raw
In reply to Subash Abhinov Kasiviswanathan
Quoted 68 lines · Subash Abhinov Kasiviswanathan
Quoted 67 lines · Bjorn Andersson
Quoted 44 lines · Alex Elder
Quoted 23 lines · Arnd Bergmann
Quoted 15 lines · Alex Elder
Quoted 13 lines · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

My question from the Nov 2018 IPA rmnet driver still stands; how does
this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is
really just a netdev talking to the IPA itself and unrelated to
net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo-
culting rmnet around just because it happens to be a net driver for a
QC SoC.

Quoted 5 lines

First, the relationship between the IPA driver and the rmnet driver
is that the IPA driver is assumed to sit between the rmnet driver
and the hardware.

Quoted 5 lines

Does this mean that IPA can only be used to back rmnet, and rmnet
can only be used on top of IPA, or can or both of them be combined
with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract
the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

Quoted 10 lines

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

But any such changes would either be years into the future or for specific devices and as such not applicable to any/most of devices on
the market now or in the coming years.

But as Arnd points out, if the software split between IPA and rmnet is
suboptimal your are encouraged to fix that.

Quoted 6 lines
Quoted 2 lines

Regards,
Bjorn

The split rmnet design was chosen because we could place rmnet over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159)
or USB.

Yeah, that's what I was looking for clarification on :) Clearly since rmnet can have many transports it should be able to be used by different HW drivers, be that qmi_wwan, IPA, and maybe even rmnet_smd.c?

Quoted 3 lines

rmnet registers a rx handler, so the rmnet packet processing itself happens in the same softirq when packets are queued to network stack by IPA.

This directly relates to the discussion about a WWAN subsystem that Johannes Berg started a couple weeks ago. IPA appears to create a netdev of its own. Is that netdev usable immediately, or does one need to create an rmnet device on top to access the default PDN?

Thanks,
Dan

#91 lore raw
In reply to Alex Elder
Quoted 28 lines · Alex Elder
Quoted 16 lines · Arnd Bergmann
Quoted 14 lines · Alex Elder
Quoted 12 lines · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

My question from the Nov 2018 IPA rmnet driver still stands; how does
this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is
really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo-
culting rmnet around just because it happens to be a net driver for a
QC SoC.

Quoted 8 lines
Quoted 4 lines

First, the relationship between the IPA driver and the rmnet driver
is that the IPA driver is assumed to sit between the rmnet driver and the hardware.

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

Quoted 8 lines

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see

To be clear I only meant cargo-culting the naming, not any functionality. Clearly IPA/rmnet/QMAP are pretty intimately connected at this point. But this goes back to whether IPA needs a netdev itself or whether you need an rmnet device created on top. If the former then I'd say no cargo-culting, if the later then it's a moot point because the device name will be rmnet%d anyway.

Dan

Quoted 82 lines

the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

Quoted 19 lines
Quoted 19 lines

Currently the modem is assumed to use QMAP protocol. This means each packet is prefixed by a (struct rmnet_map_header) structure that allows the IPA connection to be multiplexed for several logical
connections. The rmnet driver parses such messages and implements
the multiplexed network interfaces.

QMAP protocol can also be used for aggregating many small packets into a larger message. The rmnet driver implements de- aggregation
of such messages (and could probably aggregate them for TX as well).

Finally, the IPA can support checksum offload, and the rmnet driver handles providing a prepended header (for TX) and interpreting the appended trailer (for RX) if these features are enabled.

Quoted 5 lines
Quoted 3 lines

So basically, the purpose of the rmnet driver is to handle QMAP protocol connections, and right now that's what the modem provides.

Do you have any idea why this particular design was picked?

I don't really. I inherited it. Early on, when I asked about the need for QMAP I was told it was important because it offered certain features, but at that time I was somewhat new to the code and didn't have the insight to judge the merits of the design. Since then I've mostly just accepted it and concentrated on improving the IPA driver.

Quoted 8 lines

My best guess is that it evolved organically with multiple generations of hardware and software, rather than being thought out as a nice abstraction layer. If the two are tightly connected, this might mean that what we actually want here is to reintegrate the two components into a single driver with a much simpler RX and TX path that handles the checksumming and aggregation of data packets directly as it passes them from the network stack into the hardware.

In general, I agree. And Dan suggested combining the rmnet and IPA drivers into a single driver when I posted the RFC code last year. There's still the notion of switching back ends that I mentioned earlier; if that's indeed an important feature it might argue for keeping rmnet as a shim layer. But I'm really not the person to comment on this. Someone (Subash?) from Qualcomm might be able to provide better answers.

Quoted 3 lines

Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween.

My impression is the rmnet driver is a pretty thin layer, so the CPU overhead is probably not that great (though deaggregating a message is expensive). I agree with you on the flow control.

Quoted 3 lines

The intermediate buffer here acts like a router that must pass data along or randomly drop packets when the consumer can't keep up with the producer.

I haven't reviewed the rmnet code in any detail, but you may be right.

-Alex

Quoted 2 lines

Arnd

author
#92 lore raw
Quoted 8 lines · Dan Williams
Quoted 2 lines

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see

To be clear I only meant cargo-culting the naming, not any functionality. Clearly IPA/rmnet/QMAP are pretty intimately connected at this point. But this goes back to whether IPA needs a netdev itself or whether you need an rmnet device created on top. If the former then I'd say no cargo-culting, if the later then it's a moot point because the device name will be rmnet%d anyway.

OK I thought you weren't sure why rmnet was a layer at all. As I said, I didn't have a very good understanding of why it was even needed when I first started working on this.

I can't (or won't) comment right now on whether IPA needs its own netdev for rmnet to use. The IPA endpoints used for the modem network interfaces are enabled when the netdev is opened and disabled when closed. Outside of that, TX and RX are pretty much immediately passed through to the layer below or above. IPA currently has no other net device operations.

-Alex

#93 lore raw
Quoted 14 lines · Alex Elder
Quoted 12 lines · Dan Williams
Quoted 2 lines

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see

To be clear I only meant cargo-culting the naming, not any functionality. Clearly IPA/rmnet/QMAP are pretty intimately connected
at this point. But this goes back to whether IPA needs a netdev itself
or whether you need an rmnet device created on top. If the former then
I'd say no cargo-culting, if the later then it's a moot point because
the device name will be rmnet%d anyway.

Quoted 3 lines

OK I thought you weren't sure why rmnet was a layer at all. As I said, I didn't have a very good understanding of why it was even needed when I first started working on this.

No problem.

Quoted 6 lines

I can't (or won't) comment right now on whether IPA needs its own netdev for rmnet to use. The IPA endpoints used for the modem network interfaces are enabled when the netdev is opened and disabled when closed. Outside of that, TX and RX are pretty much immediately passed through to the layer below or above. IPA currently has no other net device operations.

I don't really have issues with the patchset underneath the netdev layer. I'm interested in how the various bits present themselves to userspace, which is why I am trying to tie this in with Johannes' conversation about WWAN devices, netdevs, channels, and how the various drivers present API for creating data channels that map to different modem contexts.

So let me rephrase. If the control plane has set up the default context and sent a QMI Start Network message (or the network attached the default one) and the resulting IP details are applied to the IPA netdev can things just start sending data? Or do we need to create an rmnet on top to get that working?

Dan

#94 lore raw
Quoted 14 lines
Quoted 6 lines

I can't (or won't) comment right now on whether IPA needs its own netdev for rmnet to use. The IPA endpoints used for the modem network interfaces are enabled when the netdev is opened and disabled when closed. Outside of that, TX and RX are pretty much immediately passed through to the layer below or above. IPA currently has no other net device operations.

I don't really have issues with the patchset underneath the netdev layer. I'm interested in how the various bits present themselves to userspace, which is why I am trying to tie this in with Johannes' conversation about WWAN devices, netdevs, channels, and how the various drivers present API for creating data channels that map to different modem contexts.

Quoted 7 lines

So let me rephrase. If the control plane has set up the default context and sent a QMI Start Network message (or the network attached the default one) and the resulting IP details are applied to the IPA netdev can things just start sending data? Or do we need to create an rmnet on top to get that working?

Dan

Hi Dan

All data from the hardware will have the MAP headers.
We still need to create rmnet devs over the IPA netdev and use it for the data path.
The IPA netdev will pass on the packets which it receives from the hardware and queue it to network stack where it will be intercepted by the rmnet rx handler.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#34 lore raw
In reply to Alex Elder
Quoted 26 lines · Alex Elder
Quoted 11 lines · Arnd Bergmann

On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan

- What I'm worried about most here is the flow control handling on the transmit side. The IPA driver now uses the modern BQL method to control how much data gets submitted to the hardware at any time. The rmnet driver also uses flow control using the rmnet_map_command() function, that blocks tx on the higher level device when the remote side asks us to.
I fear that doing flow control for a single physical device on two separate netdev instances is counterproductive and confuses both sides.

I understand what you're saying here, and instinctively I think you're right.

But BQL manages the *local* interface's ability to get rid of packets, whereas the QMAP flow control is initiated by the other end of the connection (the modem in this case).

With multiplexing, it's possible that one of several logical devices on the modem side has exhausted a resource and must ask the source of the data on the host side to suspend the flow. Meanwhile the other logical devices sharing the physical link might be fine, and should not be delayed by the first one.

Quoted 3 lines

It is the multiplexing itself that confuses the BQL algorithm. The abstraction obscures the *real* rates at which individual logical connections are able to transmit data.

I would assume that the real rate constantly changes, at least for wireless interfaces that are also shared with other users on the same network. BQL is meant to deal with that, at least when using a modern queuing algorithm.

Quoted 6 lines

Even if the multiple logical interfaces implemented BQL, they would not get the feedback they need directly from the IPA driver, because transmitting over the physical interface might succeed even if the logical interface on the modem side can't handle more data. So I think the flow control commands may be necessary, given multiplexing.

Can you describe what kind of multiplexing is actually going on? I'm still unclear about what we actually use multiple logical interfaces for here, and how they relate to one another.

Quoted 7 lines

The rmnet driver could use BQL, and could return NETDEV_TX_BUSY for a logical interface when its TX flow has been stopped by a QMAP command. That way the feedback for BQL on the logical interfaces would be provided in the right place.

I have no good intuition about the interaction between two layered BQL managed queues though.

Returning NETDEV_TX_BUSY is usually a bad idea as that leads to unnecessary frame drop.

I do think that using BQL and the QMAP flow command on the /same/ device would be best, as that throttles the connection when either of the two algorithms wants us to slow down.

The question is mainly which of the two devices that should be. Doing it in the ipa driver is probably easier to implement here, but ideally I think we'd only have a single queue visible to the network stack, if we can come up with a way to do that.

Quoted 12 lines
Quoted 11 lines

- I was a little confused by the location of the rmnet driver in drivers/net/ethernet/... More conventionally, I think as a protocol handler it should go into net/qmap/, with the ipa driver going into drivers/net/qmap/ipa/, similar to what we have fo ethernet, wireless, ppp, appletalk, etc.

- The rx_handler uses gro_cells, which as I understand is meant for generic tunnelling setups and takes another loop through NAPI to aggregate data from multiple queues, but in case of IPA's single-queue receive calling gro directly would be simpler and more efficient.

Quoted 3 lines

I have been planning to investigate some of the generic GRO stuff for IPA but was going to wait on that until the basic code was upstream.

That's ok, that part can easily be changed after the fact, as it does not impact the user interface or the general design.

Quoted 6 lines
Quoted 5 lines

From the overall design and the rmnet Kconfig description, it appears as though the intention as that rmnet could be a generic wrapper on top of any device, but from the implementation it seems that IPA is not actually usable that way and would always go through IPA.

Quoted 7 lines

As far as I know *nothing* upstream currently uses rmnet; the IPA driver will be the first, but as Bjorn said others seem to be on the way. I'm not sure what you mean by "IPA is not usable that way." Currently the IPA driver assumes a fixed configuration, and that configuration assumes the use of QMAP, and therefore assumes the rmnet driver is layered above it. That doesn't preclude rmnet from using a different back end.

Yes, that's what I meant above: IPA can only be used through rmnet (I wrote "through IPA", sorry for the typo), but cannot be used by itself.

Arnd

#35 lore raw
Quoted 46 lines · Arnd Bergmann
Quoted 29 lines · Alex Elder
Quoted 14 lines · Arnd Bergmann

On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan

- What I'm worried about most here is the flow control handling on the
transmit side. The IPA driver now uses the modern BQL method to control how much data gets submitted to the hardware at any
time.
The rmnet driver also uses flow control using the rmnet_map_command() function, that blocks tx on the higher level device when the remote side asks us to.
I fear that doing flow control for a single physical device on
two
separate netdev instances is counterproductive and confuses both sides.

I understand what you're saying here, and instinctively I think you're right.

But BQL manages the *local* interface's ability to get rid of packets, whereas the QMAP flow control is initiated by the other end of the connection (the modem in this case).

With multiplexing, it's possible that one of several logical devices on the modem side has exhausted a resource and must ask the source of the data on the host side to suspend the flow. Meanwhile the other logical devices sharing the physical link might be fine, and should not be delayed by the first one.

Quoted 3 lines

It is the multiplexing itself that confuses the BQL algorithm. The abstraction obscures the *real* rates at which individual logical connections are able to transmit data.

I would assume that the real rate constantly changes, at least for wireless interfaces that are also shared with other users on the same network. BQL is meant to deal with that, at least when using a modern queuing algorithm.

Quoted 6 lines

Even if the multiple logical interfaces implemented BQL, they would not get the feedback they need directly from the IPA driver, because transmitting over the physical interface might succeed even if the logical interface on the modem side can't handle more data. So I think the flow control commands may be necessary, given multiplexing.

Quoted 3 lines

Can you describe what kind of multiplexing is actually going on? I'm still unclear about what we actually use multiple logical interfaces for here, and how they relate to one another.

Each logical interface represents a different "connection" (PDP/EPS context) to the provider network with a distinct IP address and QoS. VLANs may be a suitable analogy but here they are L3+QoS.

In realistic example the main interface (say rmnet0) would be used for web browsing and have best-effort QoS. A second interface (say rmnet1) would be used for VOIP and have certain QoS guarantees from both the modem and the network itself.

QMAP can also aggregate frames for a given channel (connection/EPS/PDP context/rmnet interface/etc) to better support LTE speeds.

Dan

Quoted 61 lines
Quoted 7 lines

The rmnet driver could use BQL, and could return NETDEV_TX_BUSY for a logical interface when its TX flow has been stopped by a QMAP command. That way the feedback for BQL on the logical interfaces would be provided in the right place.

I have no good intuition about the interaction between two layered BQL managed queues though.

Returning NETDEV_TX_BUSY is usually a bad idea as that leads to unnecessary frame drop.

I do think that using BQL and the QMAP flow command on the /same/ device would be best, as that throttles the connection when either of the two algorithms wants us to slow down.

The question is mainly which of the two devices that should be. Doing it in the ipa driver is probably easier to implement here, but ideally I think we'd only have a single queue visible to the network stack, if we can come up with a way to do that.

Quoted 15 lines
Quoted 8 lines

- I was a little confused by the location of the rmnet driver in drivers/net/ethernet/... More conventionally, I think as a
protocol
handler it should go into net/qmap/, with the ipa driver going into drivers/net/qmap/ipa/, similar to what we have fo
ethernet,
wireless, ppp, appletalk, etc.

Quoted 6 lines

- The rx_handler uses gro_cells, which as I understand is meant for generic tunnelling setups and takes another loop through NAPI to aggregate data from multiple queues, but in case of IPA's single-queue receive calling gro directly would be
simpler
and more efficient.

Quoted 3 lines

I have been planning to investigate some of the generic GRO stuff for IPA but was going to wait on that until the basic code was upstream.

That's ok, that part can easily be changed after the fact, as it does not impact the user interface or the general design.

Quoted 6 lines
Quoted 5 lines

From the overall design and the rmnet Kconfig description, it appears as though the intention as that rmnet could be a generic wrapper on top of any device, but from the implementation it seems that IPA is not actually usable that way and would always go through IPA.

Quoted 7 lines

As far as I know *nothing* upstream currently uses rmnet; the IPA driver will be the first, but as Bjorn said others seem to be on the way. I'm not sure what you mean by "IPA is not usable that way." Currently the IPA driver assumes a fixed configuration, and that configuration assumes the use of QMAP, and therefore assumes the rmnet driver is layered above it. That doesn't preclude rmnet from using a different back end.

Yes, that's what I meant above: IPA can only be used through rmnet (I wrote "through IPA", sorry for the typo), but cannot be used by itself.

Arnd

#95 lore raw
In reply to Subash Abhinov Kasiviswanathan
Quoted 29 lines · Subash Abhinov Kasiviswanathan
Quoted 25 lines
Quoted 6 lines

I can't (or won't) comment right now on whether IPA needs its own netdev for rmnet to use. The IPA endpoints used for the modem network interfaces are enabled when the netdev is opened and disabled when closed. Outside of that, TX and RX are pretty much immediately passed through to the layer below or above. IPA currently has no other net device operations.

I don't really have issues with the patchset underneath the netdev layer. I'm interested in how the various bits present themselves to userspace, which is why I am trying to tie this in with Johannes' conversation about WWAN devices, netdevs, channels, and how the various
drivers present API for creating data channels that map to different
modem contexts.

So let me rephrase. If the control plane has set up the default context
and sent a QMI Start Network message (or the network attached the default one) and the resulting IP details are applied to the IPA netdev
can things just start sending data? Or do we need to create an rmnet on
top to get that working?

Quoted 1 line

Dan

Hi Dan

Quoted 9 lines

All data from the hardware will have the MAP headers. We still need to create rmnet devs over the IPA netdev and use it for
the
data path.
The IPA netdev will pass on the packets which it receives from the hardware
and queue it to network stack where it will be intercepted by the rmnet rx handler.

Ok, so IPA only needs a netdev so that rmnet has something to send/receive packets to/from? This gets even closer to the discussion in "cellular modem driver APIs - take 2" from last week.

Dan

#36 lore raw
In reply to Dan Williams
Quoted 10 lines · Dan Williams
Quoted 4 lines · Arnd Bergmann

Can you describe what kind of multiplexing is actually going on? I'm still unclear about what we actually use multiple logical interfaces for here, and how they relate to one another.

Each logical interface represents a different "connection" (PDP/EPS context) to the provider network with a distinct IP address and QoS. VLANs may be a suitable analogy but here they are L3+QoS.

Quoted 7 lines

In realistic example the main interface (say rmnet0) would be used for web browsing and have best-effort QoS. A second interface (say rmnet1) would be used for VOIP and have certain QoS guarantees from both the modem and the network itself.

QMAP can also aggregate frames for a given channel (connection/EPS/PDP context/rmnet interface/etc) to better support LTE speeds.

Thanks, that's a very helpful explanation!

Is it correct to say then that the concept of having those separate connections would be required for any proper LTE modem implementation, but the QMAP protocol (and based on that, the rmnet implementation) is Qualcomm specific and shared only among several generations of modems from that one vendor?

You mentioned the need to have a common user space interface for configuration, and if the above is true, I agree that we should try to achieve that, either by ensuring rmnet is generic enough to cover other vendors (and non-QMAP clients), or by creating a new user level interface that IPA/rmnet can be adapted to.

Arnd

#37 lore raw
Quoted 23 lines · Arnd Bergmann
Quoted 18 lines · Dan Williams
Quoted 3 lines · Arnd Bergmann

Can you describe what kind of multiplexing is actually going on? I'm still unclear about what we actually use multiple logical interfaces for here, and how they relate to one another.

Each logical interface represents a different "connection" (PDP/EPS context) to the provider network with a distinct IP address and QoS.
VLANs may be a suitable analogy but here they are L3+QoS.

In realistic example the main interface (say rmnet0) would be used for
web browsing and have best-effort QoS. A second interface (say rmnet1)
would be used for VOIP and have certain QoS guarantees from both the
modem and the network itself.

Quoted 3 lines

QMAP can also aggregate frames for a given channel (connection/EPS/PDP
context/rmnet interface/etc) to better support LTE speeds.

Quoted 8 lines

Thanks, that's a very helpful explanation!

Is it correct to say then that the concept of having those separate connections would be required for any proper LTE modem implementation,
but the QMAP protocol (and based on that, the rmnet implementation) is Qualcomm specific and shared only among several generations of modems from that one vendor?

Exactly correct. This is what Johannes is discussing in his "cellular modem APIs - take 2" thread about how this should all be organized at the driver level and I think we should figure that out before we commit to IPA-with-a-useless-netdev that requires rmnets to be created on top. That may end up being the solution but let's have that discussion.

Quoted 6 lines
Quoted 1 line

You mentioned the need to have a common user space interface

for configuration, and if the above is true, I agree that we should try
to achieve that, either by ensuring rmnet is generic enough to cover other vendors (and non-QMAP clients), or by creating a new user level interface that IPA/rmnet can be adapted to.

I would not suggest making rmnet generic; it's pretty QMAP specific (but QMAP is spoken by many many modems both SoC, USB stick, and PCIe minicard).

Instead, I think what Johannes is discussing is a better approach. A kernel WWAN framework with consistent user API that rmnet/IPA/qmi_wwan/MBIM/QMI/serial/Sierra can all implement.

That wouldn't affect the core packet processing of IPA/rmnet but instead:

1) when/how an rmnet device actually gets created on top of the IPA (or qmi_wwan) device

AND (one of these two)

a) whether IPA creates a netdev on probe

OR

b) whether there is some "WWAN device" kernel object which userspace interacts with create rmnet channels on top of IPA

Dan

author
#38 lore raw
Quoted 33 lines · Dan Williams
Quoted 23 lines · Arnd Bergmann
Quoted 18 lines · Dan Williams
Quoted 3 lines · Arnd Bergmann

Can you describe what kind of multiplexing is actually going on? I'm still unclear about what we actually use multiple logical interfaces for here, and how they relate to one another.

Each logical interface represents a different "connection" (PDP/EPS context) to the provider network with a distinct IP address and QoS.
VLANs may be a suitable analogy but here they are L3+QoS.

In realistic example the main interface (say rmnet0) would be used for
web browsing and have best-effort QoS. A second interface (say rmnet1)
would be used for VOIP and have certain QoS guarantees from both the
modem and the network itself.

Quoted 3 lines

QMAP can also aggregate frames for a given channel (connection/EPS/PDP
context/rmnet interface/etc) to better support LTE speeds.

Quoted 8 lines

Thanks, that's a very helpful explanation!

Is it correct to say then that the concept of having those separate connections would be required for any proper LTE modem implementation,
but the QMAP protocol (and based on that, the rmnet implementation) is Qualcomm specific and shared only among several generations of modems from that one vendor?

Quoted 5 lines

Exactly correct. This is what Johannes is discussing in his "cellular modem APIs - take 2" thread about how this should all be organized at the driver level and I think we should figure that out before we commit to IPA-with-a-useless-netdev that requires rmnets to be created on top. That may end up being the solution but let's have that discussion.

I looked at Johannes' message and the follow-on discussion. As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements.

HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why.

First, the layer of the IPA code that actually interacts with rmnet is very small--less than 3% if you simply do a word count of the source files. Arnd actually suggested eliminating the "ipa_netdev" files and merging their content elsewhere. This suggests two things:
- The interface to rmnet is isolated, so the effect of whatever updates are made to support a WWAN (rather than netdev) model will be focused
- The vast majority of the driver has nothing to do with that upper layer, and deals almost exclusively with managing the IPA hardware. The idea of a generic framework isn't minor, but it isn't the main focus of the IPA driver either, so I don't think it should hold it up.

Second, the IPA code has been out for review recently, and has been the subject of some detailed discussion in the past few weeks. Arnd especially has invested considerable time in review and discussion. Delaying things until after a better generic model is settled on (which I'm guessing might be on the order of months) means the IPA driver will have to be submitted for review again after that, at which point it will no longer be "fresh"; it'll be a bit like starting over.

Third, having the code upstream actually means the actual requirements for rmnet-over-IPA are clear and explicit. This might not be a huge deal, but I think it's better to devise a generic WWAN scheme that can refer to actual code than to do so with assumptions about what will work with rmnet (and others). As far as I know, the upstream rmnet has no other upstream back end; IPA will make it "real."

I support the idea of developing a generic WWAN framework, and I can assure you I'll be involved enough to perhaps be one of the first to implement a new generic scheme.

Optimistically, the IPA code itself hasn't seen much feedback for v2; maybe that means it's in good shape?

Anyway, I'd obviously like to get the IPA code accepted sooner rather than later, and I think there are good reasons to do that.

-Alex

Quoted 32 lines
Quoted 6 lines
Quoted 1 line

You mentioned the need to have a common user space interface

for configuration, and if the above is true, I agree that we should try
to achieve that, either by ensuring rmnet is generic enough to cover other vendors (and non-QMAP clients), or by creating a new user level interface that IPA/rmnet can be adapted to.

I would not suggest making rmnet generic; it's pretty QMAP specific (but QMAP is spoken by many many modems both SoC, USB stick, and PCIe minicard).

Instead, I think what Johannes is discussing is a better approach. A kernel WWAN framework with consistent user API that rmnet/IPA/qmi_wwan/MBIM/QMI/serial/Sierra can all implement.

That wouldn't affect the core packet processing of IPA/rmnet but instead:

1) when/how an rmnet device actually gets created on top of the IPA (or qmi_wwan) device

AND (one of these two)

a) whether IPA creates a netdev on probe

OR

b) whether there is some "WWAN device" kernel object which userspace interacts with create rmnet channels on top of IPA

Dan

author
#29 lore raw
In reply to Alex Elder
Quoted 34 lines · Alex Elder
Quoted 33 lines · Arnd Bergmann
Quoted 31 lines · Alex Elder
Quoted 7 lines · Arnd Bergmann
Quoted 2 lines · Alex Elder
Quoted 1 line · Dan Williams

On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote:

Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead?

No it does not mean that.

As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?)

To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features.

But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing.

Quoted 5 lines
Quoted 5 lines

Why would that require firmware changes? What I was thinking here is to turn the bits of the rmnet driver that actually do anything interesting on the headers into a library module (or a header file with inline functions) that can be called directly by the ipa driver, keeping the protocol unchanged.

OK, I follow up below, but now that I re-read this I think I misunderstood what you were saying. I now think you were just talking about having the QMAP parsing functionality provided by rmnet be put into a library that the IPA driver (and others) could use, rather than having them be two separate but stacked drivers.

That seems like a very reasonable idea, and I don't think rmnet does anything particularly special that would preclude it. But:
- I have not reviewed the rmnet driver in any detail
- It's possible this wouldn't work with other back ends
- This would also need to be considered as part of any effort to create a generic WWAN framework (though it wouldn't pose any problem as far as I can tell)

Quoted 3 lines

You know, it's possible you're right about not needing firmware changes. But it has always been my impression they would be needed. Here's why.

Now I'm just following up on the above statement.

First, what I was saying initially is that I would like to be able to configure a simple network device that does not implement any of the QMAP (rmnet) protocol. That would allow IPA to be used standalone, without any need for what the rmnet driver provides. This would suffice for the current use, because the driver currently supports *only* a single fixed-configuration data connection to the modem, and has no need for the logical channels (or aggregation) that QMAP provides.

I have not done that, because it would require both the AP side and modem side endpoints to change their configuration to *not* use QMAP. I can configure the AP endpoint, but I have no control over how the modem configures its endpoint.

Something about your question made me think I might have been misunderstanding that requirement though. I.e., I thought it might be possible for the local (AP side) endpoint to be configured to not use QMAP, regardless of what the modem side does.

I didn't inquire, but I reviewed some documents, and I now conclude that my previous understanding was correct. I can't configure an AP endpoint to not use QMAP without also having the modem configure its corresponding endpoint to not use QMAP. The AP and modem need to agree on how their respective "connected" endpoints are configured.

-Alex

Quoted 46 lines

It looks like this:

GSI Channel GSI Channel
| |

  ----------   v   -------   v   -------------
  | AP (ep)|=======| IPA |=======|(ep) Modem |
  ----------       -------       -------------

The AP and Modem each have IPA endpoints (ep), which use GSI channels, to communicate with the IPA. Each endpoint has configuration options (such as checksum offload). I *thought* that the configurations of the two endpoints need to be compatible (e.g., they need to agree on whether they're aggregating). But with your questioning I now think you may be right, that only the local endpoint's configuration matters.

I will inquire further on this. I *know* that the AP and modem exchange some information about IPA configuration, but looking more closely that looks like it's all about the configuration of shared IPA resources, not endpoints.

That said, the broader design (including the user space code) surely assumes rmnet, and I don't have any sense of what impact changing that would make. I am sure that changing it would not be well received.

-Alex

Quoted 18 lines
Quoted 8 lines
Quoted 3 lines

Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween.

My impression is the rmnet driver is a pretty thin layer, so the CPU overhead is probably not that great (though deaggregating a message is expensive). I agree with you on the flow control.

The CPU overhead I mean is not from executing code in the rmnet driver, but from passing packets through the network stack between the two drivers, i.e. adding each frame to a queue and taking it back out. I'm not sure how this ends up working in reality but from a first look it seems like we might bounce in an out of the softirq handler inbetween.

Arnd

author
#97 lore raw
In reply to Alex Elder
Quoted 24 lines · Alex Elder

This series presents the driver for the Qualcomm IP Accelerator (IPA).

This is version 2 of the series. This version has addressed almost all of the feedback received in the first version:
https://lore.kernel.org/lkml/[email protected]/
More detail is included in the individual patches, but here is a high-level summary of what's changed since then:
- Two spinlocks have been removed.
- The code for enabling and disabling endpoint interrupts has been simplified considerably, and the spinlock is no longer required
- A spinlock used when updating ring buffer pointers is no longer needed. Integers indexing the ring are used instead (and they don't even have to be atomic).
- One spinlock remains to protect list updates, but it is always acquired using spin_lock_bh() (no more irqsave).
- Information about the queueing and completion of messages is now supplied to the network stack in batches rather than one at a time.
- I/O completion handling has been simplified, with the IRQ handler now consisting mainly of disabling the interrupt and calling napi_schedule().
- Some comments have been updated and improved througout.

Quoted 1 line

What follows is the introduction supplied with v1 of the series.

Any more feedback? The only comment that I acted on is a trivial suggestion from Dave Miller: change the types for the route_virt and filter_virt fields of the ipa structure from void pointer to u64 pointer. That required no other changes to the code.

At this point I plan to post a version 3 of this series in the coming week and it will include just that one change. I might do some comment updates before then as well.

But if anyone expects to provide any additional input on the code in the near term, I can delay posting v3 until that has been addressed. If this applies to you, please let me know. (No pressure; things can always wait for v4...)

Thanks.

-Alex

Quoted 147 lines

-----

The IPA is a component present in some Qualcomm SoCs that allows network functions such as aggregation, filtering, routing, and NAT to be performed without active involvement of the main application processor (AP).

Initially, these advanced features are disabled; the IPA driver simply provides a network interface that makes the modem's LTE network available to the AP. In addition, only support for the IPA found in the Qualcomm SDM845 SoC is provided.

This code is derived from a driver developed internally by Qualcomm. A version of the original source can be seen here:
https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree
in the "drivers/platform/msm/ipa" directory. Many were involved in developing this, but the following individuals deserve explicit acknowledgement for their substantial contributions:

Abhishek Choubey
Ady Abraham
Chaitanya Pratapa
David Arinzon
Ghanim Fodi
Gidon Studinski
Ravi Gummadidala
Shihuan Liu
Skylar Chang

A version of this code was posted in November 2018 as an RFC.
https://lore.kernel.org/lkml/[email protected]/
All feedback received was addressed. The code has undergone considerable further rework since that time, and most of the "future work" described then has now been completed.

This code is available in buildable form here, based on kernel v5.2-rc1:
remote: ssh://[email protected]/people/alex.elder/linux.git
branch: ipa-v2_kernel-v5.2-rc2
75adf2ac1266 arm64: defconfig: enable build of IPA code

The branch depends on a commit now found in in net-next. It has been cherry-picked, and (in this branch) has this commit ID:
13c627b5a078 net: qualcomm: rmnet: Move common struct definitions to include
by

-Alex

Alex Elder (17):
bitfield.h: add FIELD_MAX() and field_max()
dt-bindings: soc: qcom: add IPA bindings
soc: qcom: ipa: main code
soc: qcom: ipa: configuration data
soc: qcom: ipa: clocking, interrupts, and memory
soc: qcom: ipa: GSI headers
soc: qcom: ipa: the generic software interface
soc: qcom: ipa: GSI transactions
soc: qcom: ipa: IPA interface to GSI
soc: qcom: ipa: IPA endpoints
soc: qcom: ipa: immediate commands
soc: qcom: ipa: IPA network device and microcontroller
soc: qcom: ipa: AP/modem communications
soc: qcom: ipa: support build of IPA code
MAINTAINERS: add entry for the Qualcomm IPA driver
arm64: dts: sdm845: add IPA information
arm64: defconfig: enable build of IPA code

 .../devicetree/bindings/net/qcom,ipa.yaml     |  180 ++
 MAINTAINERS                                   |    6 +
 arch/arm64/boot/dts/qcom/sdm845.dtsi          |   51 +
 arch/arm64/configs/defconfig                  |    1 +
 drivers/net/Kconfig                           |    2 +
 drivers/net/Makefile                          |    1 +
 drivers/net/ipa/Kconfig                       |   16 +
 drivers/net/ipa/Makefile                      |    7 +
 drivers/net/ipa/gsi.c                         | 1635 +++++++++++++++++
 drivers/net/ipa/gsi.h                         |  246 +++
 drivers/net/ipa/gsi_private.h                 |  148 ++
 drivers/net/ipa/gsi_reg.h                     |  376 ++++
 drivers/net/ipa/gsi_trans.c                   |  624 +++++++
 drivers/net/ipa/gsi_trans.h                   |  116 ++
 drivers/net/ipa/ipa.h                         |  131 ++
 drivers/net/ipa/ipa_clock.c                   |  297 +++
 drivers/net/ipa/ipa_clock.h                   |   52 +
 drivers/net/ipa/ipa_cmd.c                     |  377 ++++
 drivers/net/ipa/ipa_cmd.h                     |  116 ++
 drivers/net/ipa/ipa_data-sdm845.c             |  245 +++
 drivers/net/ipa/ipa_data.h                    |  267 +++
 drivers/net/ipa/ipa_endpoint.c                | 1283 +++++++++++++
 drivers/net/ipa/ipa_endpoint.h                |   97 +
 drivers/net/ipa/ipa_gsi.c                     |   48 +
 drivers/net/ipa/ipa_gsi.h                     |   49 +
 drivers/net/ipa/ipa_interrupt.c               |  279 +++
 drivers/net/ipa/ipa_interrupt.h               |   53 +
 drivers/net/ipa/ipa_main.c                    |  921 ++++++++++
 drivers/net/ipa/ipa_mem.c                     |  234 +++
 drivers/net/ipa/ipa_mem.h                     |   83 +
 drivers/net/ipa/ipa_netdev.c                  |  251 +++
 drivers/net/ipa/ipa_netdev.h                  |   24 +
 drivers/net/ipa/ipa_qmi.c                     |  402 ++++
 drivers/net/ipa/ipa_qmi.h                     |   35 +
 drivers/net/ipa/ipa_qmi_msg.c                 |  583 ++++++
 drivers/net/ipa/ipa_qmi_msg.h                 |  238 +++
 drivers/net/ipa/ipa_reg.h                     |  279 +++
 drivers/net/ipa/ipa_smp2p.c                   |  304 +++
 drivers/net/ipa/ipa_smp2p.h                   |   47 +
 drivers/net/ipa/ipa_uc.c                      |  208 +++
 drivers/net/ipa/ipa_uc.h                      |   32 +
 include/linux/bitfield.h                      |   14 +
 42 files changed, 10358 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml
 create mode 100644 drivers/net/ipa/Kconfig
 create mode 100644 drivers/net/ipa/Makefile
 create mode 100644 drivers/net/ipa/gsi.c
 create mode 100644 drivers/net/ipa/gsi.h
 create mode 100644 drivers/net/ipa/gsi_private.h
 create mode 100644 drivers/net/ipa/gsi_reg.h
 create mode 100644 drivers/net/ipa/gsi_trans.c
 create mode 100644 drivers/net/ipa/gsi_trans.h
 create mode 100644 drivers/net/ipa/ipa.h
 create mode 100644 drivers/net/ipa/ipa_clock.c
 create mode 100644 drivers/net/ipa/ipa_clock.h
 create mode 100644 drivers/net/ipa/ipa_cmd.c
 create mode 100644 drivers/net/ipa/ipa_cmd.h
 create mode 100644 drivers/net/ipa/ipa_data-sdm845.c
 create mode 100644 drivers/net/ipa/ipa_data.h
 create mode 100644 drivers/net/ipa/ipa_endpoint.c
 create mode 100644 drivers/net/ipa/ipa_endpoint.h
 create mode 100644 drivers/net/ipa/ipa_gsi.c
 create mode 100644 drivers/net/ipa/ipa_gsi.h
 create mode 100644 drivers/net/ipa/ipa_interrupt.c
 create mode 100644 drivers/net/ipa/ipa_interrupt.h
 create mode 100644 drivers/net/ipa/ipa_main.c
 create mode 100644 drivers/net/ipa/ipa_mem.c
 create mode 100644 drivers/net/ipa/ipa_mem.h
 create mode 100644 drivers/net/ipa/ipa_netdev.c
 create mode 100644 drivers/net/ipa/ipa_netdev.h
 create mode 100644 drivers/net/ipa/ipa_qmi.c
 create mode 100644 drivers/net/ipa/ipa_qmi.h
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.c
 create mode 100644 drivers/net/ipa/ipa_qmi_msg.h
 create mode 100644 drivers/net/ipa/ipa_reg.h
 create mode 100644 drivers/net/ipa/ipa_smp2p.c
 create mode 100644 drivers/net/ipa/ipa_smp2p.h
 create mode 100644 drivers/net/ipa/ipa_uc.c
 create mode 100644 drivers/net/ipa/ipa_uc.h
Reviewed-by on v2 2/17
#3 lore raw

Re: [PATCH v2 02/17] dt-bindings: soc: qcom: add IPA bindings

In reply to Alex Elder
Quoted 15 lines · Alex Elder

Add the binding definitions for the "qcom,ipa" device tree node.

---
 .../devicetree/bindings/net/qcom,ipa.yaml     | 180 ++++++++++++++++++
 1 file changed, 180 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml
Documentation/devicetree/bindings/net/qcom,ipa.yaml new file +0 -0
new file mode 100644
index 000000000000..0037fc278a61

Documentation/devicetree/bindings/net/qcom,ipa.yaml line 1

Quoted 1 line
Documentation/devicetree/bindings/net/qcom,ipa.yaml new file +1 -0
@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: GPL-2.0

New bindings are preferred to be dual GPL-2.0 and BSD-2-Clause. But that's really a decision for the submitter.

#4 lore raw

Re: [PATCH v2 02/17] dt-bindings: soc: qcom: add IPA bindings

Quoted 10 lines · Rob Herring
Quoted 9 lines · Alex Elder

Add the binding definitions for the "qcom,ipa" device tree node.

---
 .../devicetree/bindings/net/qcom,ipa.yaml     | 180 ++++++++++++++++++
 1 file changed, 180 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml
Quoted 10 lines

Documentation/devicetree/bindings/net/qcom,ipa.yaml line 1

Quoted 7 lines
Documentation/devicetree/bindings/net/qcom,ipa.yaml new file +1 -0
new file mode 100644
index 000000000000..0037fc278a61
@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: GPL-2.0

New bindings are preferred to be dual GPL-2.0 and BSD-2-Clause. But that's really a decision for the submitter.

Thanks Rob. I'll ask Qualcomm if there's any problem with doing that; I presume not. If I re-submit this with dual copyright, I will include your Reviewed-by despite the change, OK?

-Alex

Quoted 3 lines
reviewer
#39 lore raw
In reply to Alex Elder

Hi Alex, all,

Quoted 7 lines
Quoted 5 lines

Exactly correct. This is what Johannes is discussing in his "cellular modem APIs - take 2" thread about how this should all be organized at the driver level and I think we should figure that out before we commit to IPA-with-a-useless-netdev that requires rmnets to be created on top. That may end up being the solution but let's have that discussion.

I looked at Johannes' message and the follow-on discussion.

Thanks :-)

Sorry also, Dan had pointed me to this thread and the discussion, but I was travelling last week and not very reachable.

Quoted 6 lines

As I've
made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements.

I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT. Therefore, the overlap with netdevs you can really use with the network stack is pretty small?

Quoted 6 lines

HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why.

I can't really argue with those, though I would point out that the converse also holds - if we commit to this now, then we will have to actually keep the API offered by IPA/rmnet today, so we cannot actually remove the netdev again, even if we do migrate it to offer support for a WWAN framework in the future.

Quoted 5 lines

Second, the IPA code has been out for review recently, and has been the subject of some detailed discussion in the past few weeks. Arnd especially has invested considerable time in review and discussion. Delaying things until after a better generic model is settled on (which I'm guessing might be on the order of months)

I dunno if it really has to be months. I think we can cobble something together relatively quickly that addresses the needs of IPA more specifically, and then extend later?

But OTOH it may make sense to take a more paced approach and think about the details more carefully than we have over in the other thread so far.

Quoted 6 lines

Third, having the code upstream actually means the actual requirements for rmnet-over-IPA are clear and explicit. This might not be a huge deal, but I think it's better to devise a generic WWAN scheme that can refer to actual code than to do so with assumptions about what will work with rmnet (and others). As far as I know, the upstream rmnet has no other upstream back end; IPA will make it "real."

Is that really true? I had previously been told that rmnet actually does have use with a few existing drivers.

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future.

Quoted 3 lines

I support the idea of developing a generic WWAN framework, and I can assure you I'll be involved enough to perhaps be one of the first to implement a new generic scheme.

Thanks!

johannes

#40 lore raw
Quoted 10 lines · Johannes Berg
Quoted 5 lines

As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements.

I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT. Therefore, the overlap with netdevs you can really use with the network stack is pretty small?

I think Alex meant the concept of having a type of netdev with a generic user space interface for wwan and similar to a wlan device, as I understood you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/.

Quoted 7 lines
Quoted 6 lines

HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why.

Quoted 5 lines

I can't really argue with those, though I would point out that the converse also holds - if we commit to this now, then we will have to actually keep the API offered by IPA/rmnet today, so we cannot actually remove the netdev again, even if we do migrate it to offer support for a WWAN framework in the future.

Right. The interface to support rmnet might be simple enough to keep next to what becomes the generic interface, but it will always continue to be an annoyance.

Quoted 7 lines
Quoted 5 lines

Second, the IPA code has been out for review recently, and has been the subject of some detailed discussion in the past few weeks. Arnd especially has invested considerable time in review and discussion. Delaying things until after a better generic model is settled on (which I'm guessing might be on the order of months)

Quoted 6 lines

I dunno if it really has to be months. I think we can cobble something together relatively quickly that addresses the needs of IPA more specifically, and then extend later?

But OTOH it may make sense to take a more paced approach and think about the details more carefully than we have over in the other thread so far.

I would hope that as soon as we can agree on a general approach, it would also be possible to merge a minimal implementation into the kernel along with IPA. Alex already mentioned that IPA in its current state does not actually support more than one data channel, so the necessary setup for it becomes even simpler.

At the moment, the rmnet configuration in include/uapi/linux/if_link.h is almost trivial, with the three pieces of information needed being an IFLA_LINK to point to the real device (not needed if there is only one device per channel, instead of two), the IFLA_RMNET_MUX_ID setting the ID of the muxing channel (not needed if there is only one channel ?), a way to specify software bridging between channels (not useful if there is only one channel) and a few flags that I assume must match the remote end:

#define RMNET_FLAGS_INGRESS_DEAGGREGATION         (1U << 0)
#define RMNET_FLAGS_INGRESS_MAP_COMMANDS          (1U << 1)
#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4           (1U << 2)
#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4            (1U << 3)
enum {
        IFLA_RMNET_UNSPEC,
        IFLA_RMNET_MUX_ID,
        IFLA_RMNET_FLAGS,
        __IFLA_RMNET_MAX,
};
#define IFLA_RMNET_MAX  (__IFLA_RMNET_MAX - 1)
struct ifla_rmnet_flags {
        __u32   flags;
        __u32   mask;
};
Quoted 7 lines
Quoted 6 lines

Third, having the code upstream actually means the actual requirements for rmnet-over-IPA are clear and explicit. This might not be a huge deal, but I think it's better to devise a generic WWAN scheme that can refer to actual code than to do so with assumptions about what will work with rmnet (and others). As far as I know, the upstream rmnet has no other upstream back end; IPA will make it "real."

Quoted 7 lines

Is that really true? I had previously been told that rmnet actually does have use with a few existing drivers.

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future.

I would agree with that. From the code I can see no other driver including the rmnet protocol header (see the discussion about moving the header to include/linux in order to merge ipa), and I don't see any other driver referencing ETH_P_MAP either. My understanding is that any driver used by rmnet would require both, but they are all out-of-tree at the moment.

Arnd

#41 lore raw
Quoted 125 lines · Arnd Bergmann
Quoted 10 lines · Johannes Berg
Quoted 9 lines

As I've made clear before, my work on this has been focused on the IPA transport,
and some of this higher-level LTE architecture is new to me. But it
seems pretty clear that an abstracted WWAN subsystem is a good plan,
because these devices represent a superset of what a "normal" netdev
implements.

Quoted 6 lines

I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT.
Therefore, the overlap with netdevs you can really use with the network
stack is pretty small?

I think Alex meant the concept of having a type of netdev with a generic
user space interface for wwan and similar to a wlan device, as I understood
you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/.

Quoted 8 lines
Quoted 7 lines

HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's
for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why.

Quoted 8 lines

I can't really argue with those, though I would point out that the converse also holds - if we commit to this now, then we will have to
actually keep the API offered by IPA/rmnet today, so we cannot actually
remove the netdev again, even if we do migrate it to offer support for a
WWAN framework in the future.

Right. The interface to support rmnet might be simple enough to keep next to what becomes the generic interface, but it will always continue
to be an annoyance.

Quoted 9 lines
Quoted 8 lines

Second, the IPA code has been out for review recently, and has been
the subject of some detailed discussion in the past few weeks. Arnd
especially has invested considerable time in review and discussion.
Delaying things until after a better generic model is settled on (which I'm guessing might be on the order of months)

Quoted 8 lines

I dunno if it really has to be months. I think we can cobble something
together relatively quickly that addresses the needs of IPA more specifically, and then extend later?

But OTOH it may make sense to take a more paced approach and think about the details more carefully than we have over in the other thread so far.

I would hope that as soon as we can agree on a general approach, it would also be possible to merge a minimal implementation into the kernel
along with IPA. Alex already mentioned that IPA in its current state does
not actually support more than one data channel, so the necessary setup for it becomes even simpler.

At the moment, the rmnet configuration in include/uapi/linux/if_link.h
is almost trivial, with the three pieces of information needed being an IFLA_LINK to point to the real device (not needed if there is only one device per channel, instead of two), the IFLA_RMNET_MUX_ID setting the ID of the muxing channel (not needed if there is only one channel ?), a way to specify software bridging between channels (not useful if there is only one channel) and a few flags that I assume
must match the remote end:

#define RMNET_FLAGS_INGRESS_DEAGGREGATION         (1U << 0)
#define RMNET_FLAGS_INGRESS_MAP_COMMANDS          (1U << 1)
#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4           (1U << 2)
#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4            (1U << 3)
enum {
        IFLA_RMNET_UNSPEC,
        IFLA_RMNET_MUX_ID,
        IFLA_RMNET_FLAGS,
        __IFLA_RMNET_MAX,
};
#define IFLA_RMNET_MAX  (__IFLA_RMNET_MAX - 1)
struct ifla_rmnet_flags {
        __u32   flags;
        __u32   mask;
};
Quoted 17 lines
Quoted 11 lines

Third, having the code upstream actually means the actual requirements
for rmnet-over-IPA are clear and explicit. This might not be a huge
deal, but I think it's better to devise a generic WWAN scheme that
can refer to actual code than to do so with assumptions about what
will work with rmnet (and others). As far as I know, the upstream
rmnet has no other upstream back end; IPA will make it "real."

Is that really true? I had previously been told that rmnet actually does
have use with a few existing drivers.

Quoted 4 lines

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have
to actually keep the rmnet API around for all foreseeable future.

Quoted 6 lines

I would agree with that. From the code I can see no other driver including the rmnet protocol header (see the discussion about moving the header to include/linux in order to merge ipa), and I don't see any other driver referencing ETH_P_MAP either. My understanding is that any driver used by rmnet would require both, but they are all out-of-tree at the moment.

The general plan (and I believe Daniele Palmas was working on it) was to eventually make qmi_wwan use rmnet rather than its internal sysfs- based implementation. qmi_wwan and ipa are at essentially the same level and both could utilize rmnet on top.

*That's* what I'd like to see. I don't want to see two different ways to get QMAP packets to modem firmware from two different drivers that really could use the same code.

Dan

#42 lore raw
Quoted 10 lines

The general plan (and I believe Daniele Palmas was working on it) was to eventually make qmi_wwan use rmnet rather than its internal sysfs- based implementation. qmi_wwan and ipa are at essentially the same level and both could utilize rmnet on top.

*That's* what I'd like to see. I don't want to see two different ways to get QMAP packets to modem firmware from two different drivers that really could use the same code.

Dan

qmi_wwan is based on USB and is very different from the IPA interconnect though. AFAIK, they do not have much in common (apart from sending & receiving MAP packets from hardware).

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#43 lore raw
Quoted 13 lines · Subash Abhinov Kasiviswanathan
Quoted 13 lines

The general plan (and I believe Daniele Palmas was working on it) was
to eventually make qmi_wwan use rmnet rather than its internal sysfs-
based implementation. qmi_wwan and ipa are at essentially the same level and both could utilize rmnet on top.

*That's* what I'd like to see. I don't want to see two different ways
to get QMAP packets to modem firmware from two different drivers that
really could use the same code.

Quoted 6 lines
Quoted 1 line

Dan

qmi_wwan is based on USB and is very different from the IPA interconnect
though. AFAIK, they do not have much in common (apart from sending & receiving MAP packets from hardware).

That is correct, they are very different drivers but as you state they send and receive MAP packets with the other end via some closer-to- hardware protocol (USB or GSI?) than QMAP.

rmnet should handle muxing the QMAP, QoS, and aggregation and pass the resulting packet to the lower layer. That lower layer could be IPA or qmi_wwan, which in turn passes that QMAP packet to USB or GSI or whatever. This is typically how Linux handles clean abstractions between different protocol layers in drivers.

Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas for example) where the same firmware interface can be accessed via PCI, SDIO, USB, SPI, etc. The bus-specific code is self-contained and does not creep into the upper more generic parts.

Dan

#44 lore raw
Quoted 8 lines · Dan Williams

On Tue, 2019-06-11 at 10:52 -0600, Subash Abhinov Kasiviswanathan wrote:

rmnet should handle muxing the QMAP, QoS, and aggregation and pass the resulting packet to the lower layer. That lower layer could be IPA or qmi_wwan, which in turn passes that QMAP packet to USB or GSI or whatever. This is typically how Linux handles clean abstractions between different protocol layers in drivers.

Quoted 4 lines

Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas for example) where the same firmware interface can be accessed via PCI, SDIO, USB, SPI, etc. The bus-specific code is self-contained and does not creep into the upper more generic parts.

Yes, I think that is a good model. In case of libertas, we have multiple layers inheritence from the basic device (slightly different in the implementation,
but that is how it should be):

struct if_cs_card { /* pcmcia specific */

     struct lbs_private {  /* libertas specific */
           struct wireless_dev { /* 802.11 specific */
                  struct net_device {
                        struct device {
                              ...
                        };
                        ...
                  };
                  ...
           };
           ...
      };
      ...
};

The outer structure gets allocated when probing the hardware specific driver, and everything below it is implemented as direct function calls into the more generic code, or as function pointers into the more specific code.

The current rmnet model is different in that by design the upper layer (rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent in both directions, i.e. ipa has (almost) no knowledge of rmnet, and just has pointers to the other net_device:

ipa_device
net_device

rmnet_port
net_device

I understand that the rmnet model was intended to provide a cleaner abstraction, but it's not how we normally structure subsystems in Linux, and moving to a model more like how wireless_dev works would improve both readability and performance, as you describe it, it would be more like (ignoring for now the need for multiple connections):

ipa_dev
rmnet_dev
wwan_dev
net_device

Where each layer is a specialization of the next. Note: this is a common change when moving from proprietary code to upstream code. If a driver module is designed to live out of tree, there is a strong incentive to limit the number of interfaces it uses, but when it gets merged, it becomes much more flexible, as an internal interface between wwan_dev and the hardware driver(s) can be easily changed by modifying all drivers at once.

Arnd

#45 lore raw
Quoted 19 lines · Arnd Bergmann
Quoted 11 lines · Dan Williams

On Tue, 2019-06-11 at 10:52 -0600, Subash Abhinov Kasiviswanathan wrote:

rmnet should handle muxing the QMAP, QoS, and aggregation and pass the
resulting packet to the lower layer. That lower layer could be IPA or
qmi_wwan, which in turn passes that QMAP packet to USB or GSI or whatever. This is typically how Linux handles clean abstractions between different protocol layers in drivers.

Quoted 6 lines

Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas for
example) where the same firmware interface can be accessed via PCI, SDIO, USB, SPI, etc. The bus-specific code is self-contained and does
not creep into the upper more generic parts.

Quoted 5 lines

Yes, I think that is a good model. In case of libertas, we have multiple
layers inheritence from the basic device (slightly different in the implementation,
but that is how it should be):

To be clear (and I probably wasn't earlier) I wasn't talking as deep about the actual code structures as you are here but this a great discussion.

I was trying to make the point that rmnet doesn't need to care about how the QMAP packets get to the device itself; it can be pretty generic so that it can be used by IPA/qmi_wwan/rmnet_smd/etc.

Your points below are a great discussion though...

Quoted 44 lines

struct if_cs_card { /* pcmcia specific */

     struct lbs_private {  /* libertas specific */
           struct wireless_dev { /* 802.11 specific */
                  struct net_device {
                        struct device {
                              ...
                        };
                        ...
                  };
                  ...
           };
           ...
      };
      ...
};

The outer structure gets allocated when probing the hardware specific driver, and everything below it is implemented as direct function calls
into the more generic code, or as function pointers into the more specific
code.

The current rmnet model is different in that by design the upper layer
(rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent in
both directions, i.e. ipa has (almost) no knowledge of rmnet, and just
has pointers to the other net_device:

ipa_device
net_device

rmnet_port
net_device

I understand that the rmnet model was intended to provide a cleaner abstraction, but it's not how we normally structure subsystems in Linux, and moving to a model more like how wireless_dev works would improve both readability and performance, as you describe it, it would be more like (ignoring for now the need for multiple connections):

Quoted 4 lines

ipa_dev
rmnet_dev
wwan_dev
net_device

Perhaps I'm assuming too much from this diagram but this shows a 1:1 between wwan_dev and "lower" devices.

What Johannes is proposing (IIRC) is something a bit looser where a wwan_dev does not necessarily provide netdev itself, but is instead the central point that various channels (control, data, gps, sim card, etc) register with. That way the wwan_dev can provide an overall view of the WWAN device to userspace, and userspace can talk to the wwan_dev to ask the lower drivers (ipa, rmnet, etc) to create new channels (netdev, tty, otherwise) when the control channel has told the modem firmware to expect one.

For example, say you have told the firmware to create a new data channel with ID 5 via QMI (which the kernel is unaware of because it does not process higher-level QMI requests).

Perhaps (and this is all just brainstorming) then userspace asks the wwan_dev to create a new data channel with ID 5 and a certain QoS. IPA (or rmnet because that's the data channel provider for IPA) has registered callbacks to the wwan_dev, receives this request, and creates a new rmnet_dev/net_device, and then the wwan_dev passes the ifindex back to userspace so we don't have to play the dance of "match up my request with a random netlink ADD event".

The point being that since data channels aren't actually useful until the control channel agrees with the firmware that one should exist, if we have a wwan_dev that represents the entire WWAN device then we don't need the initial-but-useless net_device.

Just some thoughts; Johannes can feel free to correct me at any time :)

Quoted 7 lines

Where each layer is a specialization of the next. Note: this is a common change when moving from proprietary code to upstream code. If a driver module is designed to live out of tree, there is a strong incentive to limit the number of interfaces it uses, but when it gets merged, it becomes much more flexible, as an internal interface between wwan_dev and the hardware driver(s) can be easily changed by modifying all drivers at once.

Yep, I've seen this time and time again.

Dan

#46 lore raw
Quoted 5 lines · Dan Williams
Quoted 1 line · Arnd Bergmann

On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <[email protected]> wrote:

I was trying to make the point that rmnet doesn't need to care about how the QMAP packets get to the device itself; it can be pretty generic so that it can be used by IPA/qmi_wwan/rmnet_smd/etc.

rmnet at the moment is completely generic in that regard already, however it is implemented as a tunnel driver talking to another device rather than an abstraction layer below that driver.

Quoted 29 lines
Quoted 21 lines

The current rmnet model is different in that by design the upper layer
(rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent in
both directions, i.e. ipa has (almost) no knowledge of rmnet, and just
has pointers to the other net_device:

ipa_device
net_device

rmnet_port
net_device

I understand that the rmnet model was intended to provide a cleaner abstraction, but it's not how we normally structure subsystems in Linux, and moving to a model more like how wireless_dev works would improve both readability and performance, as you describe it, it would be more like (ignoring for now the need for multiple connections):

Quoted 4 lines

ipa_dev
rmnet_dev
wwan_dev
net_device

Perhaps I'm assuming too much from this diagram but this shows a 1:1 between wwan_dev and "lower" devices.

Quoted 8 lines

What Johannes is proposing (IIRC) is something a bit looser where a wwan_dev does not necessarily provide netdev itself, but is instead the central point that various channels (control, data, gps, sim card, etc) register with. That way the wwan_dev can provide an overall view of the WWAN device to userspace, and userspace can talk to the wwan_dev to ask the lower drivers (ipa, rmnet, etc) to create new channels (netdev, tty, otherwise) when the control channel has told the modem firmware to expect one.

Right, as I noted above, I simplified it a bit. We probably want to have multiple net_device instances for an ipa_dev, so there has to be a 1:n relationship instead of 1:1 at one of the intermediate levels, but it's not obvious which level that should be.

In theory we could even have a single net_device instance correspond to the ipa_dev, but then have multiple IP addresses bound to it, so each IP address corresponds to a channel/queue/napi_struct, but the user visible object remains a single device.

I trust that you and Johannes are more qualified than me to make the call on that point.

Arnd

reviewer
#73 lore raw
In reply to Arnd Bergmann
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Johannes Berg
Quoted 5 lines

As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements.

I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT. Therefore, the overlap with netdevs you can really use with the network stack is pretty small?

Quoted 4 lines

I think Alex meant the concept of having a type of netdev with a generic user space interface for wwan and similar to a wlan device, as I understood you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/.

I guess. It is indeed currently modelled after the stacked devices, but those regular netdevs are inherently useful by themselves, you don't *have* to tunnel or use VLANs after all.

With rmnet, the underlying netdev *isn't* useful by itself, because you're always forced to have the stacked rmnet device on top.

Quoted 13 lines
Quoted 7 lines
Quoted 6 lines

HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why.

Quoted 5 lines

I can't really argue with those, though I would point out that the converse also holds - if we commit to this now, then we will have to actually keep the API offered by IPA/rmnet today, so we cannot actually remove the netdev again, even if we do migrate it to offer support for a WWAN framework in the future.

Quoted 3 lines

Right. The interface to support rmnet might be simple enough to keep next to what becomes the generic interface, but it will always continue to be an annoyance.

Not easily, because fundamentally it requires an underlying netdev to have an ifindex, so it wouldn't just be another API to keep around (which I'd classify as an annoyance) but also a whole separate netdev that's exposed by this IPA driver, for basically this purpose only.

Quoted 4 lines
Quoted 4 lines

I dunno if it really has to be months. I think we can cobble something together relatively quickly that addresses the needs of IPA more specifically, and then extend later?

Quoted 8 lines
Quoted 2 lines

But OTOH it may make sense to take a more paced approach and think about the details more carefully than we have over in the other thread so far.

I would hope that as soon as we can agree on a general approach, it would also be possible to merge a minimal implementation into the kernel along with IPA. Alex already mentioned that IPA in its current state does not actually support more than one data channel, so the necessary setup for it becomes even simpler.

Interesting, I'm not even sure how the driver can stop multiple channels in the rmnet model?

Quoted 7 lines

At the moment, the rmnet configuration in include/uapi/linux/if_link.h is almost trivial, with the three pieces of information needed being an IFLA_LINK to point to the real device (not needed if there is only one device per channel, instead of two), the IFLA_RMNET_MUX_ID setting the ID of the muxing channel (not needed if there is only one channel ?), a way to specify software bridging between channels (not useful if there is only one channel)

I think the MUX ID is something we *would* want, and we'd probably want a channel type as well, so as to not paint ourselves into a corner where the default ends up being whatever IPA supports right now.

The software bridging is very questionable to start with, I'd advocate not supporting that at all but adding tracepoints or similar if needed for debugging instead.

Quoted 7 lines

and a few flags that I assume
must match the remote end:

#define RMNET_FLAGS_INGRESS_DEAGGREGATION         (1U << 0)
#define RMNET_FLAGS_INGRESS_MAP_COMMANDS          (1U << 1)
#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4           (1U << 2)
#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4            (1U << 3)

I don't really know about these.

Quoted 10 lines
Quoted 3 lines

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future.

I would agree with that. From the code I can see no other driver including the rmnet protocol header (see the discussion about moving the header to include/linux in order to merge ipa), and I don't see any other driver referencing ETH_P_MAP either. My understanding is that any driver used by rmnet would require both, but they are all out-of-tree at the moment.

I guess that would mean we have more work to do here, but it also means we don't have to support these interfaces forever.

I'm not *entirely* convinced though. rmnet in itself doesn't really seem to require anything from the underlying netdev, so if there's a driver that just blindly passes things through to the hardware expecting the right configuration, we wouldn't really see it this way?

OTOH, such a driver would probably blow up completely if somebody tried to use it without rmnet on top, and so it would at least have to check for ETH_P_MAP?

johannes

reviewer
#47 lore raw
In reply to Arnd Bergmann
Quoted 11 lines · Arnd Bergmann
Quoted 6 lines · Dan Williams
Quoted 1 line · Arnd Bergmann

On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <[email protected]> wrote:

I was trying to make the point that rmnet doesn't need to care about how the QMAP packets get to the device itself; it can be pretty generic so that it can be used by IPA/qmi_wwan/rmnet_smd/etc.

rmnet at the moment is completely generic in that regard already, however it is implemented as a tunnel driver talking to another device rather than an abstraction layer below that driver.

It doesn't really actually *do* much other than muck with the headers a small amount, but even that isn't really much.

You can probably implement that far more efficiently on some devices where you have a semi-decent DMA engine that at least supports S/G.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I understand that the rmnet model was intended to provide a cleaner abstraction, but it's not how we normally structure subsystems in Linux, and moving to a model more like how wireless_dev works would improve both readability and performance, as you describe it, it would be more like (ignoring for now the need for multiple connections):

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

ipa_dev
rmnet_dev
wwan_dev
net_device

Perhaps I'm assuming too much from this diagram but this shows a 1:1 between wwan_dev and "lower" devices.

I guess the fuller picture would be something like

ipa_dev
rmnet_dev
wwan_dev
net_device*

(i.e. with multiple net_devices)

Quoted 8 lines
Quoted 8 lines

What Johannes is proposing (IIRC) is something a bit looser where a wwan_dev does not necessarily provide netdev itself, but is instead the central point that various channels (control, data, gps, sim card, etc) register with. That way the wwan_dev can provide an overall view of the WWAN device to userspace, and userspace can talk to the wwan_dev to ask the lower drivers (ipa, rmnet, etc) to create new channels (netdev, tty, otherwise) when the control channel has told the modem firmware to expect one.

Yeah, that's more what I had in mind after all our discussions (will continue this below).

Quoted 9 lines

Right, as I noted above, I simplified it a bit. We probably want to have multiple net_device instances for an ipa_dev, so there has to be a 1:n relationship instead of 1:1 at one of the intermediate levels, but it's not obvious which level that should be.

In theory we could even have a single net_device instance correspond to the ipa_dev, but then have multiple IP addresses bound to it, so each IP address corresponds to a channel/queue/napi_struct, but the user visible object remains a single device.

I don't think this latter (multiple IP addresses) works well - you want a hardware specific header ("ETH_P_MAP") to carry the channel ID, without looking up the IP address and all that.

But anyway, as I alluded to above, I had something like this in mind:

driver_dev
struct device *dev (USB, PCI, ...)
net_device NA
net_device NB
tty TA
...

(I'm cutting out the rmnet layer here for now)

while having a separate that just links all the pieces together:

wwan_device W
---> dev
---> NA
---> NB
---> TA

So the driver is still responsible for creating the netdevs (or can of course delegate that to an "rmnet" library), but then all it also does is register the netdevs with the WWAN core like

wwan_add_netdev(dev, NA)

and the WWAN core would allocate the wwan_device W for this.

That way, the drivers can concentrate on providing all the necessary bits, and - crucially - even *different* drivers can end up linking to the same wwan_device. For example, if you have a modem that has a multi- function USB device, then an ethernet driver might create the netdev and a tty driver might create the control channel, but if they both agree on using the right "struct device" instance, you can still get the correct wwan_device out of it all.

And, in fact, some should then be

wwan_maybe_add_netdev(dev, N)

because the ethernet driver may not know if it attached to a modem or not, but if the control channel also attaches it's a modem for sure, with that ethernet channel attached to it.

Additionally, I'm thinking API such as

wwan_add(dev, &ops, opsdata)

that doesn't automatically attach any channels, but provides "ops" to the core to create appropriate channels. I think this latter would be something for IPA/rmnet to use, perhaps for rmnet to offer the right ops structure.

johannes

reviewer
#86 lore raw
In reply to Arnd Bergmann

On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote:

[...]

Looking at the flags again,

Quoted 1 line

#define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0)

This one I'm not sure I understand - seems weird to have such a fundamental thing as a *configuration* on the channel.

Quoted 1 line

#define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1)

Similar here? If you have flow control you probably want to use it?

Quoted 1 line

#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2)

This again looks like a hardware specific feature (ipv4 checksum)? Not sure why this is set by userspace.

Quoted 1 line

#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3)

This could be set with ethtool instead, I suppose.

johannes

reviewer
#48 lore raw
In reply to Johannes Berg
Quoted 1 line · Johannes Berg

But anyway, as I alluded to above, I had something like this in mind:

I forgot to state this here, but this was *heavily* influenced by discussions with Dan - many thanks to him.

Quoted 17 lines

driver_dev
struct device *dev (USB, PCI, ...)
net_device NA
net_device NB
tty TA
...

(I'm cutting out the rmnet layer here for now)

while having a separate that just links all the pieces together:

wwan_device W
---> dev
---> NA
---> NB
---> TA

Quoted 5 lines

So the driver is still responsible for creating the netdevs (or can of course delegate that to an "rmnet" library), but then all it also does is register the netdevs with the WWAN core like

wwan_add_netdev(dev, NA)

[snip]

So to put some more meat to this, here's an API definition for both userspace and internal APIs:

include/net/wwan.h new file +104 -0
new file mode 100644
index 000000000000..91413ec01def
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * WWAN stack interfaces
+ *
+ * Copyright (C) 2019 Intel Corporation
+ *
+ * This defines the interaction of WWAN drivers with the WWAN stack,
+ * which allows userspace configuration of sessions etc.
+ */
+#ifndef __NET_WWAN_H
+#define __NET_WWAN_H
+#include <linux/list.h>
+#include <linux/types.h>
+#include <linux/device.h>
+
+struct wwan_device {
+	u32 id;
+	void *data;
+/* private: */
+	struct list_head list;
+	struct device *dev;
+	struct wwan_device_ops *ops;
+};
+
+/**
+ * struct wwan_netdev_config - WWAN netdevice configuration
+ * @pdn: PDN identifier this netdev uses
+ */
+struct wwan_netdev_config {
+	u32 pdn;
+};
+
+/**
+ * wwan_device_ops - WWAN device operations
+ */
+struct wwan_device_ops {
+	/**
+	 * @add_netdev: Add a new netdev with the given configuration, must
+	 *	return the new netdev pointer but not call wwan_attach_netdev().
+	 */
+	struct net_device *(*add_netdev)(struct wwan_device *wwan,
+					 struct wwan_netdev_config *config);
+	/**
+	 * @remove_netdev: remove the given netdev
+	 */
+	int (*remove_netdev)(struct wwan_device *wwan, struct net_device *dev);
+
+	/*
+	 * More TBD:
+	 *  - add/remove serial port channels?
+	 *    ideally with some common (library) code to support this
+	 *    Or maybe not - serial is pretty limiting. Perhaps instead:
+	 *  - support something like AF_WWAN sockets for control data
+	 */
+};
+
+/**
+ * wwan_add - add a WWAN device without preconfigured channels
+ * @dev: underlying struct device
+ * @ops: methods to create new channels
+ * @data: data for the new WWAN device
+ *
+ * Returns: a struct wwan_device pointer, or an ERR_PTR().
+ */
+struct wwan_device *
+wwan_add(struct device *dev, struct wwan_device_ops *ops, void *data);
+
+/**
+ * wwan_remove - remove the given WWAN device
+ * @wwan: WWAN device to remove.
+ *
+ * Note that the WWAN device may not be fully removed if it still has
+ * any channels attached, but nonetheless callers must assume that the
+ * pointer is no longer valid after calling this function.
+ */
+void wwan_remove(struct wwan_device *wwan);
+
+/**
+ * wwan_attach_netdev - attach a preconfigured netdev to the WWAN device
+ * @dev: struct device underlying the WWAN device
+ * @netdev: netdev to attach
+ * @config: configuration for this netdev
+ * @tentative: mark the attachment as tentative, don't consider this as
+ *	part of a WWAN device unless other channels are attached as well.
+ *	Set this to %false in drivers that know they're a WWAN device
+ *	and to %true in generic drivers that may or may not be a WWAN
+ *	device and want to wait for other channels.
+ *
+ * Returns: a struct wwan_device pointer or ERR_PTR(). Note that a valid
+ * pointer is returned even for the tentative case.
+ *
+ * Note that there's no need to detach again, this happens automatically
+ * when the netdev is removed.
+ */
+struct wwan_device *
+wwan_attach_netdev(struct device *dev, struct net_device *netdev,
+		   struct wwan_netdev_config *config, bool tentative);
+
+/* TBD */
+struct wwan_device *
+wwan_attach_tty(struct device *dev, struct tty_port *port, bool tentative);
+void wwan_detach_tty(struct device *dev, struct tty_port *port);
+
+#endif /* __NET_WWAN_H */
include/uapi/linux/wwan.h new file +94 -0

94 lines not on this page · raw message

author
#74 lore raw
In reply to Johannes Berg
Quoted 15 lines · Johannes Berg
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Johannes Berg
Quoted 5 lines

As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements.

I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT. Therefore, the overlap with netdevs you can really use with the network stack is pretty small?

Quoted 4 lines
Quoted 4 lines

I think Alex meant the concept of having a type of netdev with a generic user space interface for wwan and similar to a wlan device, as I understood you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/.

Yes, that's pretty much what I meant by "superset." We still need netdev functionality (though not between rmnet and ipa). And it sounds like we're talking about a better framework for managing the related WWAN devices that represent logical modem connections. We're discussing more than one spot in the networking stack though, so I can see why "superset" wasn't the right word.

Quoted 6 lines

I guess. It is indeed currently modelled after the stacked devices, but those regular netdevs are inherently useful by themselves, you don't *have* to tunnel or use VLANs after all.

With rmnet, the underlying netdev *isn't* useful by itself, because you're always forced to have the stacked rmnet device on top.

Well I had mentioned earlier that I thought IPA could present just a single non-rmnet interface that could be used "directly" (i.e., without rmnet). But that would be a sort of hard-wired thing, and would not be part of the general WWAN framework under discussion.

Quoted 29 lines
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines

HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why.

Quoted 5 lines

I can't really argue with those, though I would point out that the converse also holds - if we commit to this now, then we will have to actually keep the API offered by IPA/rmnet today, so we cannot actually remove the netdev again, even if we do migrate it to offer support for a WWAN framework in the future.

Quoted 3 lines

Right. The interface to support rmnet might be simple enough to keep next to what becomes the generic interface, but it will always continue to be an annoyance.

Not easily, because fundamentally it requires an underlying netdev to have an ifindex, so it wouldn't just be another API to keep around (which I'd classify as an annoyance) but also a whole separate netdev that's exposed by this IPA driver, for basically this purpose only.

Quoted 7 lines
Quoted 6 lines

I dunno if it really has to be months. I think we can cobble something together relatively quickly that addresses the needs of IPA more specifically, and then extend later?

But OTOH it may make sense to take a more paced approach and think about the details more carefully than we have over in the other thread so far.

Quoted 8 lines
Quoted 5 lines

I would hope that as soon as we can agree on a general approach, it would also be possible to merge a minimal implementation into the kernel along with IPA. Alex already mentioned that IPA in its current state does not actually support more than one data channel, so the necessary setup for it becomes even simpler.

Interesting, I'm not even sure how the driver can stop multiple channels in the rmnet model?

Here's a little background.

The IPA driver was very large, and in an effort to have an initial driver that was more easily accepted upstream, it was carved down to support a single, very simple use case. It supports only a single channel for carrying network data, and does not expose any of the IPA's other capabilities like filtering and routing (and multiplexing).

Originally the IPA code had an IOCTL interface for adding and removing multiplexed channel IDs, but the simplified use case expected only one channel to be used. IOCTLs had to be removed to make the code acceptable for upstream, and again to simplify things, we went with a hard-wired configuration, with a single channel with an assumed set of features in use (TCP offload, basically). Once upstream, we planned to add back features in layers, including adding a netlink interface to control things like managing multiplexed channels.

The overall design assumed that the IPA connection between the modem and AP was carrying QMAP protocol though. And the rmnet driver is designed to parse and handle that, so for the design I started with the use of the rmnet driver made sense: it is a shim layer that takes care of rmnet multiplexing and aggregation (and checksum offload).

So getting back to your question, the IPA in its current form only has a single "multiplexed" channel carried over the connection between the AP and modem. Previously (and in the future) there was a way to add or remove channels.

Quoted 11 lines
Quoted 7 lines

At the moment, the rmnet configuration in include/uapi/linux/if_link.h is almost trivial, with the three pieces of information needed being an IFLA_LINK to point to the real device (not needed if there is only one device per channel, instead of two), the IFLA_RMNET_MUX_ID setting the ID of the muxing channel (not needed if there is only one channel ?), a way to specify software bridging between channels (not useful if there is only one channel)

I think the MUX ID is something we *would* want, and we'd probably want a channel type as well, so as to not paint ourselves into a corner where the default ends up being whatever IPA supports right now.

Agreed.

Quoted 3 lines

The software bridging is very questionable to start with, I'd advocate not supporting that at all but adding tracepoints or similar if needed for debugging instead.

To be honest I don't understand the connection between software bridging and debugging, but that's OK. I'm a fan of tracepoints and have always intended to make use of them in the IPA driver.

Quoted 9 lines
Quoted 7 lines

and a few flags that I assume
must match the remote end:

#define RMNET_FLAGS_INGRESS_DEAGGREGATION         (1U << 0)
#define RMNET_FLAGS_INGRESS_MAP_COMMANDS          (1U << 1)
#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4           (1U << 2)
#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4            (1U << 3)

I don't really know about these.

The hardware can aggregate multiple packets received from the modem into a single buffer, which the rmnet driver is then able to deaggregate. This feature is supposed to help performance but I've always been a little skeptical because it also comes at a cost. This is used as a flag in an rmnet (QMAP) header, which to me seems a little odd. (There should be a distinction between flags needed in a message header and flags that represent properties of a connection or channel.)

I believe the only QMAP commands are for doing essentially XON/XOFF flow control on a single channel. In the course of the e-mail discussion in the past few weeks I've come to see why that would be necessary.

The checksum offload is done differently, depending on whether it's ingress (download from modem) or egress. For egress, a header is inserted that describes what the hardware should checksum and where it should place the result. For ingress, the hardware appends a trailer that contains information about the computed checksum values. The rmnet driver is currently responsible for inserting the header and parsing the trailer.

I'm probably missing something, but I think the checksum offload could be handled by the IPA driver rather than rmnet. It seems to be an add-on that is completely independent of the multiplexing and aggregation capabilities that QMAP provides.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future.

This is because it's a user space API? If so I now understand what you mean.

As Arnd said (below) this is designed in the way out-of-tree code works and expects. I don't want to advocate for breaking that, but if a general model that supports what's required can be used, I'll adapt the IPA code to suit that.

My goal continues to be getting a baseline IPA driver accepted upstream as soon as possible, so I can then start building on that foundation.

-Alex

Quoted 21 lines
Quoted 6 lines

I would agree with that. From the code I can see no other driver including the rmnet protocol header (see the discussion about moving the header to include/linux in order to merge ipa), and I don't see any other driver referencing ETH_P_MAP either. My understanding is that any driver used by rmnet would require both, but they are all out-of-tree at the moment.

I guess that would mean we have more work to do here, but it also means we don't have to support these interfaces forever.

I'm not *entirely* convinced though. rmnet in itself doesn't really seem to require anything from the underlying netdev, so if there's a driver that just blindly passes things through to the hardware expecting the right configuration, we wouldn't really see it this way?

OTOH, such a driver would probably blow up completely if somebody tried to use it without rmnet on top, and so it would at least have to check for ETH_P_MAP?

johannes

author
#57 lore raw
In reply to Johannes Berg
Quoted 13 lines · Johannes Berg
Quoted 11 lines · Arnd Bergmann
Quoted 6 lines · Dan Williams
Quoted 1 line · Arnd Bergmann

On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <[email protected]> wrote:

I was trying to make the point that rmnet doesn't need to care about how the QMAP packets get to the device itself; it can be pretty generic so that it can be used by IPA/qmi_wwan/rmnet_smd/etc.

rmnet at the moment is completely generic in that regard already, however it is implemented as a tunnel driver talking to another device rather than an abstraction layer below that driver.

Quoted 5 lines

It doesn't really actually *do* much other than muck with the headers a small amount, but even that isn't really much.

You can probably implement that far more efficiently on some devices where you have a semi-decent DMA engine that at least supports S/G.

If it had a well-defined way of creating new channels to be multiplexed over the connection to the modem, the IPA driver (rather than the rmnet driver) could present network interfaces for each and perform the multiplexing. As I think Arnd suggested, this could at least partially be done with library code (to be shared with other "back-end" interfaces) rather than using a layered driver. This applies to aggregation, channel flow control, and checksum offload as well.

But I'm only familiar with IPA; I don't know whether the above statements make any sense for other "back-end" drivers.

Quoted 33 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I understand that the rmnet model was intended to provide a cleaner abstraction, but it's not how we normally structure subsystems in Linux, and moving to a model more like how wireless_dev works would improve both readability and performance, as you describe it, it would be more like (ignoring for now the need for multiple connections):

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

ipa_dev
rmnet_dev
wwan_dev
net_device

Perhaps I'm assuming too much from this diagram but this shows a 1:1 between wwan_dev and "lower" devices.

I guess the fuller picture would be something like

ipa_dev
rmnet_dev
wwan_dev
net_device*

(i.e. with multiple net_devices)

Quoted 8 lines
Quoted 8 lines

What Johannes is proposing (IIRC) is something a bit looser where a wwan_dev does not necessarily provide netdev itself, but is instead the central point that various channels (control, data, gps, sim card, etc) register with. That way the wwan_dev can provide an overall view of the WWAN device to userspace, and userspace can talk to the wwan_dev to ask the lower drivers (ipa, rmnet, etc) to create new channels (netdev, tty, otherwise) when the control channel has told the modem firmware to expect one.

Quoted 2 lines

Yeah, that's more what I had in mind after all our discussions (will continue this below).

This is great. The start of a more concrete discussion of the pieces that are missing...

Quoted 5 lines
Quoted 5 lines

Right, as I noted above, I simplified it a bit. We probably want to have multiple net_device instances for an ipa_dev, so there has to be a 1:n relationship instead of 1:1 at one of the intermediate levels, but it's not obvious which level that should be.

Quoted 8 lines
Quoted 4 lines

In theory we could even have a single net_device instance correspond to the ipa_dev, but then have multiple IP addresses bound to it, so each IP address corresponds to a channel/queue/napi_struct, but the user visible object remains a single device.

I don't think this latter (multiple IP addresses) works well - you want a hardware specific header ("ETH_P_MAP") to carry the channel ID, without looking up the IP address and all that.

I agree with this. It's not just multiple IP addresses for an interface, it really is multiplexed--with channel ids. It's another addressing parameter orthogonal to the IP space.

Quoted 19 lines

But anyway, as I alluded to above, I had something like this in mind:

driver_dev
struct device *dev (USB, PCI, ...)
net_device NA
net_device NB
tty TA
...

(I'm cutting out the rmnet layer here for now)

while having a separate that just links all the pieces together:

wwan_device W
---> dev
---> NA
---> NB
---> TA

Quoted 7 lines

So the driver is still responsible for creating the netdevs (or can of course delegate that to an "rmnet" library), but then all it also does is register the netdevs with the WWAN core like

wwan_add_netdev(dev, NA)

and the WWAN core would allocate the wwan_device W for this.

That would be nice. I believe you're saying that (in my case) the IPA driver creates and owns the netdevices.

But I think the IPA driver would register with the WWAN core as a "provider," and then the WWAN core would subsequently request that it instantiate netdevices to represent channels on demand (rather than registering them).

Quoted 16 lines

That way, the drivers can concentrate on providing all the necessary bits, and - crucially - even *different* drivers can end up linking to the same wwan_device. For example, if you have a modem that has a multi- function USB device, then an ethernet driver might create the netdev and a tty driver might create the control channel, but if they both agree on using the right "struct device" instance, you can still get the correct wwan_device out of it all.

And, in fact, some should then be

wwan_maybe_add_netdev(dev, N)

because the ethernet driver may not know if it attached to a modem or not, but if the control channel also attaches it's a modem for sure, with that ethernet channel attached to it.

Quoted 8 lines

Additionally, I'm thinking API such as

wwan_add(dev, &ops, opsdata)

that doesn't automatically attach any channels, but provides "ops" to the core to create appropriate channels. I think this latter would be something for IPA/rmnet to use, perhaps for rmnet to offer the right ops structure.

Yes, that's more like what I meant above. I see you're thinking as you write...

-Alex

Quoted 3 lines

johannes

#75 lore raw
In reply to Alex Elder
Quoted 8 lines · Alex Elder
Quoted 1 line · Johannes Berg

On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote:

I'm probably missing something, but I think the checksum offload could be handled by the IPA driver rather than rmnet. It seems to be an add-on that is completely independent of the multiplexing and aggregation capabilities that QMAP provides.

My best guess is that it is part of rmnet simply because this can be done in a generic way for any qmap based back-end, and rmnet was intended as the abstraction for qmap.

A better implementation of the checksumming might be to split it out into a library that is in turn used by qmap drivers. Since this should be transparent to the user interface, it can be moved there later.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future.

This is because it's a user space API? If so I now understand what you mean.

Yes, I think agreeing on the general user interface is (as usual) the one thing that has to be done as a prerequisite. I had originally hoped that by removing the ioctl interface portion of the driver, this could be avoided, but that was before I had any idea on the upper layers.

Arnd

author
#87 lore raw
In reply to Johannes Berg
Quoted 5 lines · Johannes Berg

On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote:

[...]

Looking at the flags again,

I sort of talked about this in my message a little earlier, but I now see I was partially mistaken. I thought these flags were used in messages but they're real device ("port") configuration flags.

Quoted 4 lines
Quoted 1 line

#define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0)

This one I'm not sure I understand - seems weird to have such a fundamental thing as a *configuration* on the channel.

Let me use the term "connection" to refer to the single pathway that carries data between the AP and modem. And "channel" to refer to one of several multiplexed data streams carried over that connection. (If there's better terminology, please say so; I just want to be clear in what I'm talking about.)

Deaggregation is a connection property, not a channel property. And it looks like that's exactly how it's used in the rmnet driver. The hardware is capable of aggregating QMAP packets arriving on a connection into a single buffer, so this provides a way of requesting it do that.

Quoted 3 lines
Quoted 1 line

#define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1)

Similar here? If you have flow control you probably want to use it?

I agree with that, though perhaps there are cases where it is pointless, or can't be supported, so one might want to simply *not* implement/advertise the feature. I don't know.

Quoted 8 lines
Quoted 1 line

#define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2)

This again looks like a hardware specific feature (ipv4 checksum)? Not sure why this is set by userspace.

Quoted 1 line

#define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3)

This could be set with ethtool instead, I suppose.

As I said in my earlier message, I think I concur about this. I think the IPA driver could easily hide the checksum offload capability, and if it can truly be controlled as needed using existing methods there's no need to encumber the WWAN framework with it.

-Alex

Quoted 2 lines

johannes

author
#49 lore raw
In reply to Johannes Berg
Quoted 6 lines · Johannes Berg
Quoted 1 line · Johannes Berg

But anyway, as I alluded to above, I had something like this in mind:

I forgot to state this here, but this was *heavily* influenced by discussions with Dan - many thanks to him.

Thanks for getting even more concrete with this. Code is the most concise way of describing things, once the general ideas seem to be coming together.

I'm not going to comment on the specific code bits, but I have some more general questions and comments on the design. Some of these are simply due to my lack of knowledge of how WWAN/modem interactions normally work.

First, a few terms (correct or improve as you like):
- WWAN device is a hardware device (like IPA) that presents a connection between AP and modem, and presents an interface that allows the use of that connection to be managed.
- WWAN netdevice represents a Linux network interface, with its operations and queues, etc., but implements a standardized set of WWAN-specific operations. It represents a logical
' channel whose data is multiplexed over the WWAN device.
- WWAN channel is a user space abstraction that corresponds with a WWAN netdevice (but I'm not clear on all the ways they differ or interact).
- The WWAN core is kernel code that presents abstractions for WWAN devices and netdevices, so they can be managed in a generic way. It is for configuration and communication and is not at all involved in the data path.

You're saying that the WWAN driver space calls wwan_add() to register itself as a new WWAN device.

You're also saying that a WWAN device "attaches" a WWAN netdevice, which is basically notifying the WWAN core that the new netdev/channel is available for use.
- I trust that a "tentative" attachement is necessary. But I'm not sure what makes it transition into becoming a "real" one, or how that event gets communicated.

Some questions:
- What causes a new channel to be created? Is it initiated by the WWAN device driver? Does the modem request that it get created? User space? Both?
- What causes a created channel to be removed?
- You distinguish between attaching a netdevice and (what I'll call) activating it. What causes activation?
- How are the attributes of a WWAN device or channel set, or communicated?
- Are there any attributes that are only optionally supported, and if so, how are the supported ones communicated?
- Which WWAN channel attributes must be set *before* the channel is activated, and can't be changed? Are there any that can be changed dynamically?

And while the whole point of this is to make things generic, it might be nice to have a way to implement a new feature before it can be "standardized".

Thanks.

-Alex

PS I don't want to exclude anybody but we could probably start
a different mail chain on this topic...

Quoted 7 lines
Quoted 7 lines

driver_dev
struct device *dev (USB, PCI, ...)
net_device NA
net_device NB
tty TA
...

. . .

#50 lore raw
Quoted 18 lines · Alex Elder
Quoted 7 lines · Johannes Berg
Quoted 2 lines · Johannes Berg

But anyway, as I alluded to above, I had something like this in mind:

I forgot to state this here, but this was *heavily* influenced by discussions with Dan - many thanks to him.

Thanks for getting even more concrete with this. Code is the most concise way of describing things, once the general ideas seem to be coming together.

I'm not going to comment on the specific code bits, but I have some more general questions and comments on the design. Some of these are simply due to my lack of knowledge of how WWAN/modem interactions normally work.

Quoted 11 lines

First, a few terms (correct or improve as you like):
- WWAN device is a hardware device (like IPA) that presents a connection between AP and modem, and presents an interface that allows the use of that connection to be managed.
- WWAN netdevice represents a Linux network interface, with its operations and queues, etc., but implements a standardized set of WWAN-specific operations. It represents a logical
' channel whose data is multiplexed over the WWAN device.
- WWAN channel is a user space abstraction that corresponds with a WWAN netdevice (but I'm not clear on all the ways they differ or interact).

When Johannes and I have talked about "WWAN channel" we mean a control or data or other channel. That could be QMI, AT, MBIM control, GPS, PCSC, QMAP, MBIM data, PPP TTY, DM/DIAG, CDC-ETHER, CDC-NCM, Sierra HIP, etc. Or even voice-call audio :)

A netdev is a Linux abstraction of a WWAN *data* channel, be that QMI or CDC-ETHER or whatever.

Quoted 8 lines

- The WWAN core is kernel code that presents abstractions for WWAN devices and netdevices, so they can be managed in a generic way. It is for configuration and communication and is not at all involved in the data path.

You're saying that the WWAN driver space calls wwan_add() to register itself as a new WWAN device.

Quoted 6 lines

You're also saying that a WWAN device "attaches" a WWAN netdevice, which is basically notifying the WWAN core that the new netdev/channel is available for use.
- I trust that a "tentative" attachement is necessary. But I'm not sure what makes it transition into becoming a "real" one, or how that event gets communicated.

Linux usually tries to keep drivers generic and focused; each driver is written for a specific function. For example, a USB device usually provides multiple USB interfaces which will be bound to different Linux drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc.

These drivers are often generic and we may not have enough information in one driver to know that the parent of this interface is a WWAN device. But another driver might. Since probing is asynchronous we may have cdc-acm bind to a device and provide a TTY before cdc-ether (which does know it's a WWAN) binds and provides the netdevice.

Quoted 4 lines

Some questions:
- What causes a new channel to be created? Is it initiated by the WWAN device driver? Does the modem request that it get created? User space? Both?

Either created at driver bind time in the kernel (usually control channels) or initiated by userspace when the WWAN management process has coordinated with the firmware for another channel. Honestly userspace should probably always create the netdevices (since they are always useless until userspace coordinates with the firmware about them) but that's not how things are yet.

[ A concrete example...

Assume a QMI device has an existing packet data connection which is abstracted by a netdevice on the Linux side. Now the WWAN management daemon wants to create a second packet data connection with a different APN (maybe an MMS connection, maybe a VOIP one, maybe an IPv6). It sends a WDS Start Network request to the modem firmware and receives a new QMI Packet Data Handle.

The management daemon must somehow get a netdevice associated with this new Packet Data Handle. It would ask the WWAN kernel device to create a new data channel with the PDH, and would get back the ifindex of that netdevice which it would configure with the IP that it gets from the firmware via the WDS Get Current Settings QMI request.

The WWAN device would forward the request down to IPA (or rmnet) which would then create the netdevice using the PDH as the QMAP MUX ID for that netdevice's traffic.]

Quoted 1 line

- What causes a created channel to be removed?

Driver removal, userspace WWAN daemon terminating the packet data connection which the channel represents, the modem terminating the packet data connection (eg network initiated disconnect), etc.

Quoted 2 lines

- You distinguish between attaching a netdevice and (what I'll call) activating it. What causes activation?

Can you describe what you mean by "activating"? Do you mean successfully TX/RX packets via the netdev and the outside world?

I read "attach" here as simply associating an existing netdev with the "parent" WWAN device. A purely Linux operation that is only book- keeping and may not have any interaction with the modem.

Quoted 2 lines

- How are the attributes of a WWAN device or channel set, or communicated?

Via netlink attributes when userspace asks the WWAN device to create a new channel. In the control methods I've seen, only userspace really knows the channel identifier that it and the modem have agreed on (eg what the MUX ID in the QMAP header would be, or the MBIM Session ID).

Quoted 2 lines

- Are there any attributes that are only optionally supported, and if so, how are the supported ones communicated?

Yeah, capabilities would be important here and I don't think Johannes accounted for that yet.

Quoted 3 lines

- Which WWAN channel attributes must be set *before* the channel is activated, and can't be changed? Are there any that can be changed dynamically?

I would assume userspace must pass the agreed identifier (QMUX ID, MBIM session ID, etc) when creating the channel and that wouldn't change. I think a world where you can dynamically change the MUX ID/SessionID/etc is a more complicated one.

Things like QoS could change but I don't recall if modems allow that; eg does a +CGEQOS (or equivalent QMI WDS LTE QoS Parameters request) take effect while the bearer is active, or is it only respected on bearer creation?

Quoted 3 lines

And while the whole point of this is to make things generic, it might be nice to have a way to implement a new feature before it can be "standardized".

That would be nice, but I'd rather have the conversation about if/how to standardize things before they make it into the kernel and have their API set in stone... which is how we ended up with 5 ways of doing the same thing already.

Dan

Quoted 16 lines

Thanks.

-Alex

PS I don't want to exclude anybody but we could probably start
a different mail chain on this topic...

Quoted 7 lines
Quoted 7 lines

driver_dev
struct device *dev (USB, PCI, ...)
net_device NA
net_device NB
tty TA
...

. . .

reviewer
#55 lore raw
In reply to Alex Elder

Just to add to Dan's response, I think he's captured our discussions and thoughts well.

Quoted 1 line

First, a few terms (correct or improve as you like):

Thanks for defining, we don't do that nearly often enough.

Quoted 3 lines

- WWAN device is a hardware device (like IPA) that presents a connection between AP and modem, and presents an interface that allows the use of that connection to be managed.

Yes. But I was actually thinking of a "wwan_dev" to be a separate structure, not *directly* owned by a single driver and used to represent the hardware like a (hypothetical) "struct ipa_dev".

Quoted 4 lines

- WWAN netdevice represents a Linux network interface, with its operations and queues, etc., but implements a standardized set of WWAN-specific operations. It represents a logical
' channel whose data is multiplexed over the WWAN device.

I'm not sure I'd asy it has much WWAN-specific operations? But yeah, I guess it might.

Quoted 3 lines

- WWAN channel is a user space abstraction that corresponds with a WWAN netdevice (but I'm not clear on all the ways they differ or interact).

As Dan said, this could be a different abstraction than a netdevice, like a TTY, etc.

Quoted 7 lines

- The WWAN core is kernel code that presents abstractions for WWAN devices and netdevices, so they can be managed in a generic way. It is for configuration and communication and is not at all involved in the data path.

You're saying that the WWAN driver space calls wwan_add() to register itself as a new WWAN device.

Assuming it knows that it is in fact a WWAN device, like IPA.

Quoted 6 lines

You're also saying that a WWAN device "attaches" a WWAN netdevice, which is basically notifying the WWAN core that the new netdev/channel is available for use.
- I trust that a "tentative" attachement is necessary. But I'm not sure what makes it transition into becoming a "real" one, or how that event gets communicated.

I think Dan explained this one well. This wasn't actually on my radar until he pointed it out.

Really this only exists with USB devices that appear as multiple functions (ethernet, tty, ...) but still represent a single WWAN device, with each function not necessarily being aware of that since it's just a function driver.

Hopefully at least one of the function drivers will be able to figure it out, and then we can combine all of the functions into the WWAN device abstraction.

[snip - Dan's explanations are great]

Quoted 3 lines · Dan also
Quoted 3 lines

I read "attach" here as simply associating an existing netdev with the "parent" WWAN device. A purely Linux operation that is only book- keeping and may not have any interaction with the modem.

Now I'm replying out of thread, but yes, that's what I had in mind. What I meant by attaching (in this case) is just that you actually mark that it is (or might be, if tentatively attached) part of a WWAN device.

Quoted 2 lines

- Are there any attributes that are only optionally supported, and if so, how are the supported ones communicated?

As Dan said, good point. I hadn't really considered that for now. I sort of know that we need it, but for the sake of simplicity decided to elide it for now. I'm just not sure what really are needed, and netlink attributes make adding them (and discovering the valid ones) pretty easy in the future, when a need arises.

Quoted 3 lines

- Which WWAN channel attributes must be set *before* the channel is activated, and can't be changed? Are there any that can be changed dynamically?

It's a good question. I threw a "u32 pdn" in there, but I'm not actually sure that's what you *really* need?

Maybe the modem and userspace just agree on some arbitrary "session identifier"? Dan mentions "MUX ID" or "MBIM Session ID", maybe there really is no good general term for this and we should just call it a "session identifier" and agree that it depends on the control protocol (MBIM vs. QMI vs. ...)?

Quoted 3 lines

And while the whole point of this is to make things generic, it might be nice to have a way to implement a new feature before it can be "standardized".

Not sure I understand this?

FWIW, I actually came to this because we want to upstream a driver for an Intel modem, but ... can't really make up our mind on whether or not to use VLAN tags, something like rmnet (but we obviously cannot use rmnet, so that'd be another vendor specific interface like rmnet), or sysfs, or any of the other methods we have today ... :-)

johannes

reviewer
#58 lore raw
In reply to Alex Elder
Quoted 4 lines · Alex Elder

If it had a well-defined way of creating new channels to be multiplexed over the connection to the modem, the IPA driver (rather than the rmnet driver) could present network interfaces for each and perform the multiplexing.

Right. That's what I was thinking of.

I actually expect this to fare much better going forward with 5G around the corner, since you'll want to eventually take advantage of multi- queue TX or RSS for RX, queue size control and what not, as speeds increase.

Because of these things I think the whole "layered netdev" approach is actually *wrong* rather than just inconvenient.

In particular, in the Intel driver, you're going to have multiple hardware queues, one for each ongoing session. This means that multiplexing it over a layered netdev like rmnet or something like VLAN (which some driver does too) actually prevents us from doing this properly - it means we need to implement ndo_select_queue() and multiple queues on the underlying netdev etc., and then we no longer have the ability to use actual multi-queue. It becomes messy very very quickly.

Quoted 5 lines

As I think Arnd
suggested, this could at least partially be done with library code (to be shared with other "back-end" interfaces) rather than using a layered driver. This applies to aggregation, channel flow control, and checksum offload as well.

Right.

Quoted 2 lines

But I'm only familiar with IPA; I don't know whether the above statements make any sense for other "back-end" drivers.

I think they do, in different ways. Intel probably wouldn't have a library - there isn't actually much of a MUX header because there are different hardware queues for the different sessions.

Quoted 2 lines

This is great. The start of a more concrete discussion of the pieces that are missing...

:-)

I think I said before - it should be pretty easy to mold some code around the API I proposed there and have something reasonably functional soon.

Quoted 2 lines

That would be nice. I believe you're saying that (in my case) the IPA driver creates and owns the netdevices.

Yes.

Quoted 4 lines

But I think the IPA driver would register with the WWAN core as a "provider," and then the WWAN core would subsequently request that it instantiate netdevices to represent channels on demand (rather than registering them).

Yeah, I guess you could call it that way.

Really there are two possible ways (and they intersect to some extent).

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

Now, it's not clear to me where IPA actually falls, because so far we've been talking about the IPA driver only as providing *netdevs*, not any control channels, so I'm not actually sure where the control channel is.

For the Intel device, however, the control channel is definitely provided by exactly the same driver as the data channels (netdevs).

"provider" is a good word, and in fact the Intel driver would also be a provider for a GNSS channel (TBD how to represent, a tty?), one or multiple debug/tracing channels, data channels (netdevs), AT command channels (mbim, ...?) (again tbd how to represent, ttys?), etc.

What I showed in the header files I posted so far was the provider only having "data channel" ops (create/remove a netdev) but for each channel type we either want a new method there, or we just change the method to be something like

	int (*create_channel)(..., enum wwan_chan_type chan_type, ...);

and simply require that the channel is attached to the wwan device with the representation-specific call (wwan_attach_netdev, wwan_attach_tty, ...).

This is a bit less comfortable because then it's difficult to know what was actually created upon the request, so it's probably better to have different methods for the different types of representations (like I had
- add_netdev, add_tty, ...).

Note also that I said "representation-specific", while passing a "channel type", so for this we'd actually need a convention on what channel type has what kind of representation, which again gets awkward. Better to make it explicit.

(And even then, we might be able to let userspace have some control, e.g. the driver might be able to create a debug channel as both a TTY or something else)

johannes

reviewer
#76 lore raw
In reply to Alex Elder
Quoted 21 lines · Alex Elder
Quoted 15 lines · Johannes Berg
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Johannes Berg
Quoted 5 lines

As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements.

I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT. Therefore, the overlap with netdevs you can really use with the network stack is pretty small?

Quoted 4 lines
Quoted 4 lines

I think Alex meant the concept of having a type of netdev with a generic user space interface for wwan and similar to a wlan device, as I understood you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/.

Quoted 6 lines

Yes, that's pretty much what I meant by "superset." We still need netdev functionality (though not between rmnet and ipa). And it sounds like we're talking about a better framework for managing the related WWAN devices that represent logical modem connections. We're discussing more than one spot in the networking stack though, so I can see why "superset" wasn't the right word.

Right, ok, gotcha. I was focused (here at least) much more on the netdevs, rather than the whole "superset" of functionality :-)

Quoted 11 lines
Quoted 6 lines

I guess. It is indeed currently modelled after the stacked devices, but those regular netdevs are inherently useful by themselves, you don't *have* to tunnel or use VLANs after all.

With rmnet, the underlying netdev *isn't* useful by itself, because you're always forced to have the stacked rmnet device on top.

Well I had mentioned earlier that I thought IPA could present just a single non-rmnet interface that could be used "directly" (i.e., without rmnet). But that would be a sort of hard-wired thing, and would not be part of the general WWAN framework under discussion.

Oh, I guess I didn't see that (got to the thread late), but is that actually useful? It doesn't seem very useful to me since you can't actually do anything there.

Quoted 1 line

Here's a little background.

That's great to have :-) Let me read and comment/ask questions.

Quoted 5 lines

The IPA driver was very large, and in an effort to have an initial driver that was more easily accepted upstream, it was carved down to support a single, very simple use case. It supports only a single channel for carrying network data, and does not expose any of the IPA's other capabilities like filtering and routing (and multiplexing).

Ok. But it *does* use (or even require using) rmnet, so it has multiple channels in a sense, no?

Quoted 3 lines

Originally the IPA code had an IOCTL interface for adding and removing multiplexed channel IDs, but the simplified use case expected only one channel to be used.

What did those channels do? Create different netdevs? Something else?

Quoted 6 lines

IOCTLs had to be removed to make the code acceptable
for upstream, and again to simplify things, we went with a hard-wired configuration, with a single channel with an assumed set of features in use (TCP offload, basically). Once upstream, we planned to add back features in layers, including adding a netlink interface to control things like managing multiplexed channels.

Right, ok.

Quoted 5 lines

The overall design assumed that the IPA connection between the modem and AP was carrying QMAP protocol though. And the rmnet driver is designed to parse and handle that, so for the design I started with the use of the rmnet driver made sense: it is a shim layer that takes care of rmnet multiplexing and aggregation (and checksum offload).

Sure, I can't really disagree. It's just that we have an ongoing discussion separately about whether or not rmnet really makes sense itself, mostly starting from our interest in supporting the Intel modem, and realizing that we have like 5 or 6 different driver-specific interfaces of doing the same thing.

Quoted 4 lines

So getting back to your question, the IPA in its current form only has a single "multiplexed" channel carried over the connection between the AP and modem. Previously (and in the future) there was a way to add or remove channels.

What would those channels do?

I've not really been very clear with the differentiation between a channel and what's multiplexed inside of the channel.

Using the terminology you defined in your other mail, are you saying that IPA (originally) allowed multiple *connections* to the device, or is there basically just one connection, with multiple (QMAP-muxed) *channels* on top of it?

If the latter, why did IPA need ioctls, rather than rmnet?

Quoted 6 lines
Quoted 3 lines

The software bridging is very questionable to start with, I'd advocate not supporting that at all but adding tracepoints or similar if needed for debugging instead.

To be honest I don't understand the connection between software bridging and debugging, but that's OK.

It's a mess. Basically, AFAICT, the only use for the rmnet bridging is in fact debugging. What it does, again AFAICT, is mirror out all the rmnet packets to the bridge if you attach it to a bridge, so that then you can attach another netdev to the bridge and forward all the rmnet packets to another system for debugging.

It's a very weird way of doing this, IMHO.

Quoted 2 lines

I'm a fan of tracepoints
and have always intended to make use of them in the IPA driver.

:-)

Quoted 3 lines

The hardware can aggregate multiple packets received from the modem into a single buffer, which the rmnet driver is then able to deaggregate.

Right, I gathered that much, but I'm not really sure I see why userspace would even be allowed to control this? Either the device is doing it or not, but the driver is going to have to cope either way?

Quoted 6 lines

This feature is supposed to help performance
but I've always been a little skeptical because it also comes at a cost. This is used as a flag in an rmnet (QMAP) header, which to me seems a little odd. (There should be a distinction between flags needed in a message header and flags that represent properties of a connection or channel.)

I'm not going to comment on the QMAP protocol, I know nothing about it :-)

Quoted 4 lines

I believe the only QMAP commands are for doing essentially XON/XOFF flow control on a single channel. In the course of the e-mail discussion in the past few weeks I've come to see why that would be necessary.

It does make sense, because you only have a single hardware (DMA) channel in these cases, so you implement flow control in software on top.

(As I said before, the Intel modem uses different hardware channels for different sessions, so doesn't need something like this - the hardware ring just fills up and there's your flow control)

Quoted 8 lines

The checksum offload is done differently, depending on whether it's ingress (download from modem) or egress. For egress, a header is inserted that describes what the hardware should checksum and where it should place the result. For ingress, the hardware appends a trailer that contains information about the computed checksum values. The rmnet driver is currently responsible for inserting the header and parsing the trailer.

Sure, makes sense, but again - if you can negotiate with the modem on whether it's going to do RX CSUM offload, you can tell it with ethtool (assuming you do have a netdev per channel, I guess), and for TX CSUM you can just have the header or not depending on whether you want it and the modem is capable of it.

Quoted 5 lines

I'm probably missing something, but I think the checksum offload could be handled by the IPA driver rather than rmnet. It seems to be an add-on that is completely independent of the multiplexing and aggregation capabilities that QMAP provides.

Agree.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future.

This is because it's a user space API? If so I now understand what you mean.

Yes.

Quoted 8 lines

As Arnd said (below) this is designed in the way out-of-tree code works and expects. I don't want to advocate for breaking that, but if a general model that supports what's required can be used, I'll adapt the IPA code to suit that.

My goal continues to be getting a baseline IPA driver accepted upstream as soon as possible, so I can then start building on that foundation.

Yeah. My goal is actually the same, but for the Intel driver, but I don't have much code yet (it's being cleaned up now) :-)

johannes

reviewer
#88 lore raw
In reply to Alex Elder
Quoted 1 line · Alex Elder

Deaggregation is a connection property, not a channel property.

That'd make sense, yes.

Quoted 2 lines

And it looks like that's exactly how it's used in the rmnet driver.

Yeah, I think you're right. I got confused by the whole use of "port" there, but it seems like "port" actually refers to the underlying netdev.

Which is really strange too, btw, because you configure the "port" to agg/non-agg when you add a new channel to it ... So it seems like it's part of the channel configuration, when it's not!

Anyway, I think for now we could probably live with not having this configurable for the IPA driver, and if it *does* need to be configurable, it seems like it should be a driver configuration, not a channel configuration - so something like a debugfs hook if you really just need to play with it for performance testing, or a module parameter, or something else?

Or even, in the WWAN framework, a knob that we provide there for the WWAN device, rather than for the (newly created) channel.

Quoted 11 lines

The hardware is capable of aggregating QMAP packets arriving on a connection into a single buffer, so this provides a way of requesting it do that.

Quoted 3 lines
Quoted 1 line

#define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1)

Similar here? If you have flow control you probably want to use it?

I agree with that, though perhaps there are cases where it is pointless, or can't be supported, so one might want to simply *not* implement/advertise the feature. I don't know.

Sure, but then that's likely something the driver would need to know, not necessarily userspace?

johannes

#77 lore raw
In reply to Johannes Berg
Quoted 12 lines · Johannes Berg
Quoted 5 lines · Alex Elder

On 6/17/19 6:28 AM, Johannes Berg wrote:
So getting back to your question, the IPA in its current form only has a single "multiplexed" channel carried over the connection between the AP and modem. Previously (and in the future) there was a way to add or remove channels.

What would those channels do?

I've not really been very clear with the differentiation between a channel and what's multiplexed inside of the channel.

Quoted 8 lines

Using the terminology you defined in your other mail, are you saying that IPA (originally) allowed multiple *connections* to the device, or is there basically just one connection, with multiple (QMAP-muxed) *channels* on top of it?

If the latter, why did IPA need ioctls, rather than rmnet?

From my understanding, the ioctl interface would create the lower

netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The software bridging is very questionable to start with, I'd advocate not supporting that at all but adding tracepoints or similar if needed for debugging instead.

To be honest I don't understand the connection between software bridging and debugging, but that's OK.

Quoted 7 lines

It's a mess. Basically, AFAICT, the only use for the rmnet bridging is in fact debugging. What it does, again AFAICT, is mirror out all the rmnet packets to the bridge if you attach it to a bridge, so that then you can attach another netdev to the bridge and forward all the rmnet packets to another system for debugging.

It's a very weird way of doing this, IMHO.

My understanding for this was that the idea is to use it for connecting bridging between distinct hardware devices behind
ipa: if IPA drives both a USB-ether gadget and the 5G modem, you can use to talk to Linux running rmnet, but you can also use rmnet to provide fast usb tethering to 5g and bypass the rest of the network stack. That again may have been a wrong guess on my part.

Quoted 5 lines
Quoted 4 lines

I believe the only QMAP commands are for doing essentially XON/XOFF flow control on a single channel. In the course of the e-mail discussion in the past few weeks I've come to see why that would be necessary.

Quoted 7 lines

It does make sense, because you only have a single hardware (DMA) channel in these cases, so you implement flow control in software on top.

(As I said before, the Intel modem uses different hardware channels for different sessions, so doesn't need something like this - the hardware ring just fills up and there's your flow control)

ipa definitely has multiple hardware queues, and the Alex' driver does implement the data path on those, just not the configuration to enable them.

Guessing once more, I suspect the the XON/XOFF flow control was a workaround for the fact that rmnet and ipa have separate queues. The hardware channel on IPA may fill up, but user space talks to rmnet and still add more frames to it because it doesn't know IPA is busy.

Another possible explanation would be that this is actually forwarding state from the base station to tell the driver to stop sending data over the air.

Arnd

#59 lore raw
In reply to Johannes Berg
Quoted 11 lines · Johannes Berg

On Tue, 2019-06-18 at 08:45 -0500, Alex Elder wrote:

Really there are two possible ways (and they intersect to some extent).

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

I would hope we can simplify this to expect only the second model, where you have a 'struct device' corresponding to hardware and the driver for it creates one wwan_device that user space talks to.

Clearly the multi-function device hardware has to be handled somehow, but it would seem much cleaner in the long run to do that using a special workaround rather than putting this into the core interface.

E.g. have a driver that lets you create a wwan_device by passing netdev and a tty chardev into a configuration interface, and from that point on use the generic wwan abstraction.

Quoted 3 lines

Now, it's not clear to me where IPA actually falls, because so far we've been talking about the IPA driver only as providing *netdevs*, not any control channels, so I'm not actually sure where the control channel is.

The IPA driver today only handles the data path, because Alex removed the control channel. IPA is the driver that needs to talk to the hardware, both for data and control when finished. rmnet is a pure software construct that also contains both a data and control side and is designed to be independent of the lower hardware.

Arnd

reviewer
#60 lore raw
Quoted 8 lines · Arnd Bergmann
Quoted 8 lines

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

Quoted 7 lines
Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

I would hope we can simplify this to expect only the second model, where you have a 'struct device' corresponding to hardware and the driver for it creates one wwan_device that user space talks to.

I'm not sure.

Fundamentally, we have drivers in Linux for the ethernet part, for the TTY part, and for whatever other part might be in a given USB multi- function device.

Quoted 3 lines

Clearly the multi-function device hardware has to be handled somehow, but it would seem much cleaner in the long run to do that using a special workaround rather than putting this into the core interface.

I don't think it really makes the core interface much more complex or difficult though, and it feels easier than writing a completely different USB driver yet again for all these devices?

As far as I understand from Dan, sometimes they really are no different from a generic USB TTY and a generic USB ethernet, except you know that if those show up together it's a modem.

Quoted 3 lines

E.g. have a driver that lets you create a wwan_device by passing netdev and a tty chardev into a configuration interface, and from that point on use the generic wwan abstraction.

Yeah, but where do you hang that driver? Maybe the TTY function is actually a WWAN specific USB driver, but the ethernet is something generic that can also work with pure ethernet USB devices, and it's difficult to figure out how to tie those together. The modules could load in completely different order, or even the ethernet module could load but the TTY one doesn't because it's not configured, or vice versa.

Quoted 9 lines
Quoted 3 lines

Now, it's not clear to me where IPA actually falls, because so far we've been talking about the IPA driver only as providing *netdevs*, not any control channels, so I'm not actually sure where the control channel is.

The IPA driver today only handles the data path, because Alex removed the control channel. IPA is the driver that needs to talk to the hardware, both for data and control when finished. rmnet is a pure software construct that also contains both a data and control side and is designed to be independent of the lower hardware.

I'd actually be interested in what the control path should be like.

Is it also muxed on QMAP in the same way?

johannes

#61 lore raw
Quoted 38 lines · Johannes Berg
Quoted 7 lines · Arnd Bergmann
Quoted 7 lines

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

Quoted 7 lines
Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

I would hope we can simplify this to expect only the second model, where you have a 'struct device' corresponding to hardware and the driver for it creates one wwan_device that user space talks to.

I'm not sure.

Fundamentally, we have drivers in Linux for the ethernet part, for the TTY part, and for whatever other part might be in a given USB multi- function device.

Quoted 3 lines

Clearly the multi-function device hardware has to be handled somehow, but it would seem much cleaner in the long run to do that using a special workaround rather than putting this into the core interface.

I don't think it really makes the core interface much more complex or difficult though, and it feels easier than writing a completely different USB driver yet again for all these devices?

As far as I understand from Dan, sometimes they really are no different from a generic USB TTY and a generic USB ethernet, except you know that if those show up together it's a modem.

Quoted 3 lines

E.g. have a driver that lets you create a wwan_device by passing netdev and a tty chardev into a configuration interface, and from that point on use the generic wwan abstraction.

Quoted 6 lines

Yeah, but where do you hang that driver? Maybe the TTY function is actually a WWAN specific USB driver, but the ethernet is something generic that can also work with pure ethernet USB devices, and it's difficult to figure out how to tie those together. The modules could load in completely different order, or even the ethernet module could load but the TTY one doesn't because it's not configured, or vice versa.

That was more or less my point: The current drivers exist, but don't lean themselves to fitting into a new framework, so maybe the best answer is not to try fitting them.

To clarify: I'm not suggesting to write new USB drivers for these at all, but instead keep three parts that are completely unaware of each other

a)  a regular netdevice driver
b)  a regular tty driver
c)  the new wwan subsystem that expects a device to be created

from a hardware driver but knows nothing of a) and b)

To connect these together, we need one glue driver that implements the wwan_device and talks to a) and b) as the hardware. There are many ways to do that. One way would be to add a tty ldisc driver. A small user space helper opens the chardev, sets the ldisc and then uses an ldisc specific ioctl command to create a wwan device by passing an identifier of the netdevice and then exits.

Quoted 1 line

From that point on, you have a wwan device like any other.

Arnd

reviewer
#78 lore raw
In reply to Arnd Bergmann
Quoted 6 lines · Arnd Bergmann

From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code.

Huh. But if rmnet has muxing, and IPA supports that, why would you ever need multiple lower netdevs?

Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

The software bridging [...]

My understanding for this was that the idea is to use it for connecting bridging between distinct hardware devices behind
ipa: if IPA drives both a USB-ether gadget and the 5G modem, you can use to talk to Linux running rmnet, but you can also use rmnet to provide fast usb tethering to 5g and bypass the rest of the network stack. That again may have been a wrong guess on my part.

Hmm. Interesting. It didn't really look to me like that, but I'm really getting lost in the code. Anyway, it seems weird, because then you'd just bridge the upper netdev with the other ethernet and don't need special logic? And I don't see how the ethernet headers would work with this now.

Quoted 3 lines

ipa definitely has multiple hardware queues, and the Alex' driver does implement the data path on those, just not the configuration to enable them.

OK, but perhaps you don't actually have enough to use one for each session?

Quoted 9 lines

Guessing once more, I suspect the the XON/XOFF flow control was a workaround for the fact that rmnet and ipa have separate queues. The hardware channel on IPA may fill up, but user space talks to rmnet and still add more frames to it because it doesn't know IPA is busy.

Another possible explanation would be that this is actually forwarding state from the base station to tell the driver to stop sending data over the air.

Yeah, but if you actually have a hardware queue per upper netdev then you don't really need this - you just stop the netdev queue when the hardware queue is full, and you have flow control automatically.

So I really don't see any reason to have these messages going back and forth unless you plan to have multiple sessions muxed on a single hardware queue.

And really, if you don't mux multiple sessions onto a single hardware queue, you don't need a mux header either, so it all adds up :-)

johannes

reviewer
#62 lore raw
In reply to Arnd Bergmann
Quoted 18 lines · Arnd Bergmann
Quoted 6 lines

Yeah, but where do you hang that driver? Maybe the TTY function is actually a WWAN specific USB driver, but the ethernet is something generic that can also work with pure ethernet USB devices, and it's difficult to figure out how to tie those together. The modules could load in completely different order, or even the ethernet module could load but the TTY one doesn't because it's not configured, or vice versa.

That was more or less my point: The current drivers exist, but don't lean themselves to fitting into a new framework, so maybe the best answer is not to try fitting them.

To clarify: I'm not suggesting to write new USB drivers for these at all, but instead keep three parts that are completely unaware of each other

a)  a regular netdevice driver
b)  a regular tty driver
c)  the new wwan subsystem that expects a device to be created

from a hardware driver but knows nothing of a) and b)

Quoted 7 lines

To connect these together, we need one glue driver that implements the wwan_device and talks to a) and b) as the hardware. There are many ways to do that. One way would be to add a tty ldisc driver. A small user space helper opens the chardev, sets the ldisc and then uses an ldisc specific ioctl command to create a wwan device by passing an identifier of the netdevice and then exits. From that point on, you have a wwan device like any other.

Well, ok. I don't think it'd really work that way ("passing an identifier of the netdevice") because you could have multiple netdevices, but I see what you mean in principle.

It seems to me though that this is far more complex than what I'm proposing? What I'm proposing there doesn't even need any userspace involvement, as long as all the pieces are in the different sub-drivers, they'd fall out automatically.

And realistically, the wwan_device falls out anyway at some point, the only question is if we really make one specific driver be the "owner" of it. I'm suggesting that we don't, and just make its lifetime depend on the links to parts it has (unless something like IPA actually wants to be an owner).

johannes

#79 lore raw
In reply to Johannes Berg
Quoted 9 lines · Johannes Berg
Quoted 6 lines · Arnd Bergmann

From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code.

Quoted 4 lines

Huh. But if rmnet has muxing, and IPA supports that, why would you ever need multiple lower netdevs?

From my reading of the code, there is always exactly a 1:1 relationship

between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ decapsulation of the qmap data and forwards it to the ipa netdev, which then just passes data through between a hardware queue and its netdevice.

[side note: on top of that, rmnet also does "aggregation", which may
be a confusing term that only means transferring multiple frames at once]

Quoted 6 lines
Quoted 3 lines

ipa definitely has multiple hardware queues, and the Alex' driver does implement the data path on those, just not the configuration to enable them.

OK, but perhaps you don't actually have enough to use one for each session?

I'm lacking the terminology here, but what I understood was that the netdev and queue again map to a session.

Quoted 10 lines
Quoted 9 lines

Guessing once more, I suspect the the XON/XOFF flow control was a workaround for the fact that rmnet and ipa have separate queues. The hardware channel on IPA may fill up, but user space talks to rmnet and still add more frames to it because it doesn't know IPA is busy.

Another possible explanation would be that this is actually forwarding state from the base station to tell the driver to stop sending data over the air.

Quoted 7 lines

Yeah, but if you actually have a hardware queue per upper netdev then you don't really need this - you just stop the netdev queue when the hardware queue is full, and you have flow control automatically.

So I really don't see any reason to have these messages going back and forth unless you plan to have multiple sessions muxed on a single hardware queue.

Sure, I definitely understand what you mean, and I agree that would be the right way to do it. All I said is that this is not how it was done in rmnet (this was again my main concern about the rmnet design after I learned it was required for ipa) ;-)

Arnd

reviewer
#80 lore raw
Quoted 11 lines · Arnd Bergmann
Quoted 9 lines · Johannes Berg
Quoted 6 lines · Arnd Bergmann

From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code.

Quoted 8 lines
Quoted 2 lines

Huh. But if rmnet has muxing, and IPA supports that, why would you ever need multiple lower netdevs?

From my reading of the code, there is always exactly a 1:1 relationship between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ decapsulation of the qmap data and forwards it to the ipa netdev, which then just passes data through between a hardware queue and its netdevice.

I'll take your word for it. Seems very odd, given that the whole point of the QMAP header seems to be ... muxing?

Quoted 3 lines

[side note: on top of that, rmnet also does "aggregation", which may
be a confusing term that only means transferring multiple frames at once]

Right, but it's not all that much interesting in the context of this discussion.

Quoted 4 lines

Sure, I definitely understand what you mean, and I agree that would be the right way to do it. All I said is that this is not how it was done in rmnet (this was again my main concern about the rmnet design after I learned it was required for ipa) ;-)

:-)

Well, I guess though if the firmware wants us to listen to those on/off messages we'll have to do that one way or the other.

Oh. Maybe it's just *because* rmnet is layered on top, and thus you fundamentally cannot do flow control the way I described - not because you have multiple session on the same hardware ring, but because you abstracted the hardware ring away too much ...

johannes

#63 lore raw
In reply to Johannes Berg
Quoted 7 lines · Johannes Berg

On Tue, 2019-06-18 at 22:33 +0200, Arnd Bergmann wrote:

It seems to me though that this is far more complex than what I'm proposing? What I'm proposing there doesn't even need any userspace involvement, as long as all the pieces are in the different sub-drivers, they'd fall out automatically.

Quoted 5 lines

And realistically, the wwan_device falls out anyway at some point, the only question is if we really make one specific driver be the "owner" of it. I'm suggesting that we don't, and just make its lifetime depend on the links to parts it has (unless something like IPA actually wants to be an owner).

My feeling so far is that having the wwan_device be owned by a device gives a nicer abstraction model that is also simpler for the common case. A device driver like ipa would end up with a probe() function that does does wwan_device_alloc/wwan_device_register, corresponding to alloc_etherdev/register_netdev, and then communicates through callbacks.

I agree the compound device case would get more complex by shoehorning it into this model, but that can be a valid tradeoff if it's the exceptional case rather than the common one.

Arnd

#81 lore raw
In reply to Arnd Bergmann
Quoted 11 lines · Arnd Bergmann
Quoted 9 lines · Johannes Berg
Quoted 6 lines · Arnd Bergmann

From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code.

Quoted 9 lines
Quoted 2 lines

Huh. But if rmnet has muxing, and IPA supports that, why would you ever need multiple lower netdevs?

From my reading of the code, there is always exactly a 1:1 relationship between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ decapsulation of the qmap data and forwards it to the ipa netdev, which then just passes data through between a hardware queue and its netdevice.

There is a n:1 relationship between rmnet and IPA. rmnet does the de-muxing to multiple netdevs based on the mux id in the MAP header for RX packets and vice versa.

Quoted 8 lines

[side note: on top of that, rmnet also does "aggregation", which may
be a confusing term that only means transferring multiple frames at once]

Quoted 4 lines
Quoted 3 lines

ipa definitely has multiple hardware queues, and the Alex' driver does implement the data path on those, just not the configuration to enable them.

Quoted 24 lines
Quoted 2 lines

OK, but perhaps you don't actually have enough to use one for each session?

I'm lacking the terminology here, but what I understood was that the netdev and queue again map to a session.

Quoted 17 lines
Quoted 9 lines

Guessing once more, I suspect the the XON/XOFF flow control was a workaround for the fact that rmnet and ipa have separate queues. The hardware channel on IPA may fill up, but user space talks to rmnet and still add more frames to it because it doesn't know IPA is busy.

Another possible explanation would be that this is actually forwarding state from the base station to tell the driver to stop sending data over the air.

Yeah, but if you actually have a hardware queue per upper netdev then you don't really need this - you just stop the netdev queue when the hardware queue is full, and you have flow control automatically.

So I really don't see any reason to have these messages going back and forth unless you plan to have multiple sessions muxed on a single hardware queue.

Hardware may flow control specific PDNs (rmnet interfaces) based on QoS - not necessarily only in case of hardware queue full.

Quoted 6 lines

Sure, I definitely understand what you mean, and I agree that would be the right way to do it. All I said is that this is not how it was done in rmnet (this was again my main concern about the rmnet design after I learned it was required for ipa) ;-)

Arnd

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#82 lore raw
Quoted 24 lines · Subash Abhinov Kasiviswanathan
Quoted 15 lines · Arnd Bergmann
Quoted 9 lines · Johannes Berg
Quoted 6 lines · Arnd Bergmann

From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code.

Quoted 3 lines

Huh. But if rmnet has muxing, and IPA supports that, why would you ever
need multiple lower netdevs?

Quoted 6 lines

From my reading of the code, there is always exactly a 1:1 relationship between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ decapsulation of the qmap data and forwards it to the ipa netdev, which then just passes data through between a hardware queue and its netdevice.

Quoted 3 lines

There is a n:1 relationship between rmnet and IPA. rmnet does the de-muxing to multiple netdevs based on the mux id in the MAP header for RX packets and vice versa.

Oh, so you mean that even though IPA supports multiple channels and multiple netdev instances for a physical device, all the rmnet devices end up being thrown into a single channel in IPA?

What are the other channels for in IPA? I understand that there is one channel for commands that is separate, while the others are for network devices, but that seems to make no sense if we only use a single channel for rmnet data.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Yeah, but if you actually have a hardware queue per upper netdev then you don't really need this - you just stop the netdev queue when the hardware queue is full, and you have flow control automatically.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

So I really don't see any reason to have these messages going back and forth unless you plan to have multiple sessions muxed on a single hardware queue.

Hardware may flow control specific PDNs (rmnet interfaces) based on QoS -
not necessarily only in case of hardware queue full.

Right, I guess that makes sense if everything ends up in a single queue in IPA.

Arnd

#83 lore raw
Quoted 4 lines
Quoted 3 lines

There is a n:1 relationship between rmnet and IPA. rmnet does the de-muxing to multiple netdevs based on the mux id in the MAP header for RX packets and vice versa.

Quoted 9 lines

Oh, so you mean that even though IPA supports multiple channels and multiple netdev instances for a physical device, all the rmnet devices end up being thrown into a single channel in IPA?

What are the other channels for in IPA? I understand that there is one channel for commands that is separate, while the others are for network devices, but that seems to make no sense if we only use a single channel for rmnet data.

AFAIK, the other channels are for use cases like tethering. There is only a single channel which is used for RX data which is then de-muxed using rmnet.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#64 lore raw
In reply to Arnd Bergmann
Quoted 26 lines · Arnd Bergmann
Quoted 7 lines · Johannes Berg

On Tue, 2019-06-18 at 22:33 +0200, Arnd Bergmann wrote:
It seems to me though that this is far more complex than what I'm proposing? What I'm proposing there doesn't even need any userspace involvement, as long as all the pieces are in the different sub- drivers,
they'd fall out automatically.

Quoted 9 lines

And realistically, the wwan_device falls out anyway at some point, the
only question is if we really make one specific driver be the "owner" of
it. I'm suggesting that we don't, and just make its lifetime depend on
the links to parts it has (unless something like IPA actually wants to
be an owner).

My feeling so far is that having the wwan_device be owned by a device gives a nicer abstraction model that is also simpler for the common case. A device driver like ipa would end up with a probe() function that does does wwan_device_alloc/wwan_device_register, corresponding to alloc_etherdev/register_netdev, and then communicates through callbacks.

Quoted 3 lines

I agree the compound device case would get more complex by shoehorning it into this model, but that can be a valid tradeoff if it's the exceptional case rather than the common one.

In my experience, the compound device model is by far the most prevalent for regular Linux distros or anything *not* running on an SoC with an integrated modem.

But it's also quite common for Android, no? drivers/net/ethernet/msm/ has rmnet and IPA ethernet drivers while arch/arm/mach-msm/ has various SMD-related control channel drivers like smd_tty.c and smd_qmi.c and smd_nmea.c. At least that's how I remember older SMD-based devices being in the 8xxx and 9xxx time.

Ideally those setups can benefit from this framework as well, without having to write entirely new composite drivers for those devices.

Dan

#84 lore raw
In reply to Subash Abhinov Kasiviswanathan
Quoted 14 lines · Subash Abhinov Kasiviswanathan
Quoted 4 lines
Quoted 3 lines

There is a n:1 relationship between rmnet and IPA. rmnet does the de-muxing to multiple netdevs based on the mux id in the MAP header for RX packets and vice versa.

Quoted 9 lines

Oh, so you mean that even though IPA supports multiple channels and multiple netdev instances for a physical device, all the rmnet devices end up being thrown into a single channel in IPA?

What are the other channels for in IPA? I understand that there is one channel for commands that is separate, while the others are for network devices, but that seems to make no sense if we only use a single channel for rmnet data.

Quoted 3 lines

AFAIK, the other channels are for use cases like tethering. There is only a single channel which is used for RX data which is then de-muxed using rmnet.

That seems odd, since tethering is no different than any other data channel in QMI, just that it may have a different APN and QoS guarantees.

Dan

#98 lore raw

Re: [PATCH v2 05/17] soc: qcom: ipa: clocking, interrupts, and memory

In reply to Alex Elder

Hello

Review

Quoted 4 lines · On Thu, 30 May 2019 20:54:01 -0700 (PDT) Alex Elder wrote:
(quoted diff) +4 -0
+
+void ipa_interrupt_teardown(struct ipa_interrupt *interrupt)
+{
+	free_irq(interrupt->irq, interrupt);

Looks like that there is something missed.

	kfree(interrupt);
Quoted 1 line

+}

Hillf

author
#51 lore raw
In reply to Dan Williams
Quoted 6 lines · Dan Williams
Quoted 5 lines · Alex Elder
Quoted 4 lines · Johannes Berg
Quoted 2 lines · Johannes Berg

But anyway, as I alluded to above, I had something like this in mind:

Sorry for the delay. There's a lot here to go through, and with each message the picture is (slowly) getting a bit clearer for me. Still, there are some broad tradeoffs to consider and I think we need to get a little more specific again. I'm going to start a new thread (or rather re-subject a response to the very first one) that tries to do a fresh start that takes into account the discussion so far.

I will also be talking with some people inside Qualcomm (including Subash) soon to make sure we don't miss any requirements or insights they know of that I don't realize are important.

But before I send anything new I'm going to respond to a few things.

Quoted 24 lines
Quoted 12 lines
Quoted 2 lines

I forgot to state this here, but this was *heavily* influenced by discussions with Dan - many thanks to him.

Thanks for getting even more concrete with this. Code is the most concise way of describing things, once the general ideas seem to be coming together.

I'm not going to comment on the specific code bits, but I have some more general questions and comments on the design. Some of these are simply due to my lack of knowledge of how WWAN/modem interactions normally work.

Quoted 11 lines

First, a few terms (correct or improve as you like):
- WWAN device is a hardware device (like IPA) that presents a connection between AP and modem, and presents an interface that allows the use of that connection to be managed.
- WWAN netdevice represents a Linux network interface, with its operations and queues, etc., but implements a standardized set of WWAN-specific operations. It represents a logical
' channel whose data is multiplexed over the WWAN device.
- WWAN channel is a user space abstraction that corresponds with a WWAN netdevice (but I'm not clear on all the ways they differ or interact).

Quoted 7 lines

When Johannes and I have talked about "WWAN channel" we mean a control or data or other channel. That could be QMI, AT, MBIM control, GPS, PCSC, QMAP, MBIM data, PPP TTY, DM/DIAG, CDC-ETHER, CDC-NCM, Sierra HIP, etc. Or even voice-call audio :)

A netdev is a Linux abstraction of a WWAN *data* channel, be that QMI or CDC-ETHER or whatever.

I think I now understand this. My only focus with the IPA driver has been the network data driver. I'll go into more detail later but I now see that there are other entities on a WWAN device that do not require a netdev.

Quoted 15 lines
Quoted 8 lines

- The WWAN core is kernel code that presents abstractions for WWAN devices and netdevices, so they can be managed in a generic way. It is for configuration and communication and is not at all involved in the data path.

You're saying that the WWAN driver space calls wwan_add() to register itself as a new WWAN device.

Quoted 6 lines

You're also saying that a WWAN device "attaches" a WWAN netdevice, which is basically notifying the WWAN core that the new netdev/channel is available for use.
- I trust that a "tentative" attachement is necessary. But I'm not sure what makes it transition into becoming a "real" one, or how that event gets communicated.

Quoted 4 lines

Linux usually tries to keep drivers generic and focused; each driver is written for a specific function. For example, a USB device usually provides multiple USB interfaces which will be bound to different Linux drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc.

So USB has some attributes similar to what we're talking about here. But if I'm not mistaken we want some sort of an overall management scheme as well.

Quoted 5 lines

These drivers are often generic and we may not have enough information in one driver to know that the parent of this interface is a WWAN device. But another driver might. Since probing is asynchronous we may have cdc-acm bind to a device and provide a TTY before cdc-ether (which does know it's a WWAN) binds and provides the netdevice.

Is this why Johannes wanted to have a "maybe attach" method?

I don't like the "maybe" API unless there's no other way to do it.

Instead I think it would be better for the probing driver to register with a whatever the WWAN core is, and then have the WWAN core be responsible for pulling things all together when it receives a request to do so. I.e., something in user space should request that a registered data interface be brought up, and at that time everything "knows" it's implemented as part of a WWAN device.

Quoted 8 lines
Quoted 4 lines

Some questions:
- What causes a new channel to be created? Is it initiated by the WWAN device driver? Does the modem request that it get created? User space? Both?

Either created at driver bind time in the kernel (usually control channels) or initiated by userspace when the WWAN management process has coordinated with the firmware for another channel. Honestly

So maybe:
- Hardware probe detects a WWAN device
- The drivers that detect the WWAN device register it with the WWAN core code.
- A control channel is instantiated at/before the time the WWAN device is registered
- Something in user space should manage the bring-up of any other things on the WWAN device thereafter

Quoted 3 lines

userspace should probably always create the netdevices (since they are always useless until userspace coordinates with the firmware about them) but that's not how things are yet.

That's too bad. How hard would that be to change?

Quoted 15 lines

[ A concrete example...

Assume a QMI device has an existing packet data connection which is abstracted by a netdevice on the Linux side. Now the WWAN management daemon wants to create a second packet data connection with a different APN (maybe an MMS connection, maybe a VOIP one, maybe an IPv6). It sends a WDS Start Network request to the modem firmware and receives a new QMI Packet Data Handle.

The management daemon must somehow get a netdevice associated with this new Packet Data Handle. It would ask the WWAN kernel device to create a new data channel with the PDH, and would get back the ifindex of that netdevice which it would configure with the IP that it gets from the firmware via the WDS Get Current Settings QMI request.

Quoted 3 lines

The WWAN device would forward the request down to IPA (or rmnet) which would then create the netdevice using the PDH as the QMAP MUX ID for that netdevice's traffic.]

OK yes I'm following this now. I appreciate the example.

Quoted 5 lines
Quoted 1 line

- What causes a created channel to be removed?

Driver removal, userspace WWAN daemon terminating the packet data connection which the channel represents, the modem terminating the packet data connection (eg network initiated disconnect), etc.

OK this is as I expected. Driver (or device) removal is somewhat obvious, but you're confirming user space might request it as well.

Quoted 5 lines
Quoted 2 lines

- You distinguish between attaching a netdevice and (what I'll call) activating it. What causes activation?

Can you describe what you mean by "activating"? Do you mean successfully TX/RX packets via the netdev and the outside world?

Johannes mentioned an API to "maybe attach" a device. That begs the question of what happens if this request does *not* attach. Does the attach request have to be made again, or is it done automatically with a notification, or something else?

So by "activation" I was trying to refer to the notion of this subsequent successful attach.

Quoted 3 lines

I read "attach" here as simply associating an existing netdev with the "parent" WWAN device. A purely Linux operation that is only book- keeping and may not have any interaction with the modem.

If that's the case I would want the "activation" to be a separate step. The attach would do the bookkeeping, and generally shouldn't fail. An attached interface would be brought up ("activated") separately and might fail if things aren't quite ready yet.

Quoted 7 lines
Quoted 2 lines

- How are the attributes of a WWAN device or channel set, or communicated?

Via netlink attributes when userspace asks the WWAN device to create a new channel. In the control methods I've seen, only userspace really knows the channel identifier that it and the modem have agreed on (eg what the MUX ID in the QMAP header would be, or the MBIM Session ID).

Yes, that's the way it's worked for rmnet and IPA. Previously it was IOCTL requests but it's currently hard-wired.

Quoted 15 lines
Quoted 2 lines

- Are there any attributes that are only optionally supported, and if so, how are the supported ones communicated?

Yeah, capabilities would be important here and I don't think Johannes accounted for that yet.

Quoted 3 lines

- Which WWAN channel attributes must be set *before* the channel is activated, and can't be changed? Are there any that can be changed dynamically?

I would assume userspace must pass the agreed identifier (QMUX ID, MBIM session ID, etc) when creating the channel and that wouldn't change. I think a world where you can dynamically change the MUX ID/SessionID/etc is a more complicated one.

Quoted 4 lines

Things like QoS could change but I don't recall if modems allow that; eg does a +CGEQOS (or equivalent QMI WDS LTE QoS Parameters request) take effect while the bearer is active, or is it only respected on bearer creation?

You are speaking in a language I'm only now coming to understand. I think the point of my question is clear though--I think both static and dynamic attributes need to be taken into account.

Quoted 8 lines
Quoted 3 lines

And while the whole point of this is to make things generic, it might be nice to have a way to implement a new feature before it can be "standardized".

That would be nice, but I'd rather have the conversation about if/how to standardize things before they make it into the kernel and have their API set in stone... which is how we ended up with 5 ways of doing the same thing already.

Agreed.

-Alex

Quoted 19 lines

Dan

Quoted 16 lines

Thanks.

-Alex

PS I don't want to exclude anybody but we could probably start
a different mail chain on this topic...

Quoted 7 lines
Quoted 7 lines

driver_dev
struct device *dev (USB, PCI, ...)
net_device NA
net_device NB
tty TA
...

. . .

author
#56 lore raw
In reply to Johannes Berg
Quoted 7 lines · Johannes Berg

Just to add to Dan's response, I think he's captured our discussions and thoughts well.

Quoted 1 line

First, a few terms (correct or improve as you like):

Thanks for defining, we don't do that nearly often enough.

Quoted 7 lines
Quoted 3 lines

- WWAN device is a hardware device (like IPA) that presents a connection between AP and modem, and presents an interface that allows the use of that connection to be managed.

Yes. But I was actually thinking of a "wwan_dev" to be a separate structure, not *directly* owned by a single driver and used to represent the hardware like a (hypothetical) "struct ipa_dev".

I think you're talking about creating a coordination interface that allows multiple drivers to interact with a WWAN device, which might implement several independent features.

Quoted 7 lines
Quoted 4 lines

- WWAN netdevice represents a Linux network interface, with its operations and queues, etc., but implements a standardized set of WWAN-specific operations. It represents a logical
' channel whose data is multiplexed over the WWAN device.

I'm not sure I'd asy it has much WWAN-specific operations? But yeah, I guess it might.

I want to withdraw this notion of a "WWAN netdevice"...

Quoted 6 lines
Quoted 3 lines

- WWAN channel is a user space abstraction that corresponds with a WWAN netdevice (but I'm not clear on all the ways they differ or interact).

As Dan said, this could be a different abstraction than a netdevice, like a TTY, etc.

Right, I get that now.

. . .

Quoted 13 lines
Quoted 3 lines

- Which WWAN channel attributes must be set *before* the channel is activated, and can't be changed? Are there any that can be changed dynamically?

It's a good question. I threw a "u32 pdn" in there, but I'm not actually sure that's what you *really* need?

Maybe the modem and userspace just agree on some arbitrary "session identifier"? Dan mentions "MUX ID" or "MBIM Session ID", maybe there really is no good general term for this and we should just call it a "session identifier" and agree that it depends on the control protocol (MBIM vs. QMI vs. ...)?

Quoted 5 lines
Quoted 3 lines

And while the whole point of this is to make things generic, it might be nice to have a way to implement a new feature before it can be "standardized".

Not sure I understand this?

I'm talking about a way to experiment with new functionality in a way that's explicitly not part of the interface. But doing that isn't necessary and it's probably not a good idea anyway.

Quoted 5 lines

FWIW, I actually came to this because we want to upstream a driver for an Intel modem, but ... can't really make up our mind on whether or not to use VLAN tags, something like rmnet (but we obviously cannot use rmnet, so that'd be another vendor specific interface like rmnet), or sysfs, or any of the other methods we have today ... :-)

OK cool then we have some common needs. Let's get this defined so
we can use it for both!

-Alex

Quoted 3 lines

johannes

author
#65 lore raw
In reply to Johannes Berg
Quoted 8 lines · Johannes Berg
Quoted 4 lines · Alex Elder

If it had a well-defined way of creating new channels to be multiplexed over the connection to the modem, the IPA driver (rather than the rmnet driver) could present network interfaces for each and perform the multiplexing.

Right. That's what I was thinking of.

. . .

Quoted 7 lines
Quoted 4 lines

But I think the IPA driver would register with the WWAN core as a "provider," and then the WWAN core would subsequently request that it instantiate netdevices to represent channels on demand (rather than registering them).

Yeah, I guess you could call it that way.

Quoted 8 lines

Really there are two possible ways (and they intersect to some extent).

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

I *think* this model makes the most sense. But at this point it would take very little to convince me otherwise... (And then I saw Arnd's message advocating the other one, unfortunately...)

Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

What I don't like about this is that it's more monolithic. It seems better to have the low-level IPA or Intel modem driver (or any other driver that can support communication between the AP and WWAN device) present communication paths that other function- specific drivers can attach to and use.

Quoted 3 lines

Now, it's not clear to me where IPA actually falls, because so far we've been talking about the IPA driver only as providing *netdevs*, not any control channels, so I'm not actually sure where the control channel is.

There is user space code that handles all of this, and as far as I can tell, parts of it will always remain proprietary.

Quoted 2 lines

For the Intel device, however, the control channel is definitely provided by exactly the same driver as the data channels (netdevs).

I do see the need for a control interface. But I suspect it would *overlap* with what you describe and might need to be more general and/or extensible. Are there control channels specific to use for a modem--like a "modem control interface" or something? Is there something broader, like "this WWAN device supports functions A, and B with protocols X, Y; please open a connection to A with protocol X." Do both exist? I'm just trying to contain whatever a "control channel" really represents, and what it would be associated with.

Quoted 4 lines

"provider" is a good word, and in fact the Intel driver would also be a provider for a GNSS channel (TBD how to represent, a tty?), one or multiple debug/tracing channels, data channels (netdevs), AT command channels (mbim, ...?) (again tbd how to represent, ttys?), etc.

Yes, this is much clearer to me now.

Quoted 10 lines

What I showed in the header files I posted so far was the provider only having "data channel" ops (create/remove a netdev) but for each channel type we either want a new method there, or we just change the method to be something like

	int (*create_channel)(..., enum wwan_chan_type chan_type, ...);

and simply require that the channel is attached to the wwan device with the representation-specific call (wwan_attach_netdev, wwan_attach_tty, ...).

Or maybe have the WWAN device present interfaces with attributes, and have drivers that are appropriate for each interface attach to only the ones they recognize they support.

-Alex

Quoted 16 lines

This is a bit less comfortable because then it's difficult to know what was actually created upon the request, so it's probably better to have different methods for the different types of representations (like I had
- add_netdev, add_tty, ...).

Note also that I said "representation-specific", while passing a "channel type", so for this we'd actually need a convention on what channel type has what kind of representation, which again gets awkward. Better to make it explicit.

(And even then, we might be able to let userspace have some control, e.g. the driver might be able to create a debug channel as both a TTY or something else)

johannes

author
#85 lore raw
In reply to Johannes Berg
Quoted 7 lines · Johannes Berg
Quoted 6 lines · Alex Elder
Quoted 5 lines · Johannes Berg
Quoted 4 lines · Arnd Bergmann
Quoted 1 line · Johannes Berg
Quoted 1 line

As I've made clear before, my work on this has been focused on the IPA transport,

. . .

Quoted 8 lines
Quoted 5 lines

The IPA driver was very large, and in an effort to have an initial driver that was more easily accepted upstream, it was carved down to support a single, very simple use case. It supports only a single channel for carrying network data, and does not expose any of the IPA's other capabilities like filtering and routing (and multiplexing).

Ok. But it *does* use (or even require using) rmnet, so it has multiple channels in a sense, no?

Yes. It's a multiplexing protocol, supporting one *or more* channels.

Quoted 5 lines
Quoted 3 lines

Originally the IPA code had an IOCTL interface for adding and removing multiplexed channel IDs, but the simplified use case expected only one channel to be used.

What did those channels do? Create different netdevs? Something else

I don't know. The code I started with only supported the use of one channel, but assumed the use of rmnet anyway (for aggregation and checksum offload most likely).

. . .

Quoted 7 lines
Quoted 4 lines

So getting back to your question, the IPA in its current form only has a single "multiplexed" channel carried over the connection between the AP and modem. Previously (and in the future) there was a way to add or remove channels.

What would those channels do?

Quoted 7 lines

I've not really been very clear with the differentiation between a channel and what's multiplexed inside of the channel.

Using the terminology you defined in your other mail, are you saying that IPA (originally) allowed multiple *connections* to the device, or is there basically just one connection, with multiple (QMAP-muxed) *channels* on top of it?

One connection, with the ability to have multiple multiplexed channels.

Quoted 1 line

If the latter, why did IPA need ioctls, rather than rmnet?

The "full" IPA driver supported a lot more than what is being proposed right now. The strategy for getting support upstream was to drastically reduce the size of the driver by focusing on a single use case: providing a netdev data interface served by the modem.

But even to support that, the IPA driver needed to allow user space to identify certain resources that need to be used for implementing that netdev, or configuring whether certain features (e.g. download checksum) were to be used.

. . .

Quoted 7 lines
Quoted 3 lines

The hardware can aggregate multiple packets received from the modem into a single buffer, which the rmnet driver is then able to deaggregate.

Right, I gathered that much, but I'm not really sure I see why userspace would even be allowed to control this? Either the device is doing it or not, but the driver is going to have to cope either way?

Maybe because doing aggregation or not is a policy decision? And/or a tunable parameter? There might be a more appropriate way to do this.

. . .

Quoted 6 lines
Quoted 3 lines

My goal continues to be getting a baseline IPA driver accepted upstream as soon as possible, so I can then start building on that foundation.

Yeah. My goal is actually the same, but for the Intel driver, but I don't have much code yet (it's being cleaned up now) :-)

Well then I guess I'll beat you to it (or I *hope* I do)...

-Alex

Quoted 3 lines

johannes

author
#89 lore raw
In reply to Johannes Berg
Quoted 1 line · Johannes Berg

On Tue, 2019-06-18 at 09:00 -0500, Alex Elder wrote:

. . .

Quoted 9 lines

Anyway, I think for now we could probably live with not having this configurable for the IPA driver, and if it *does* need to be configurable, it seems like it should be a driver configuration, not a channel configuration - so something like a debugfs hook if you really just need to play with it for performance testing, or a module parameter, or something else?

Or even, in the WWAN framework, a knob that we provide there for the WWAN device, rather than for the (newly created) channel.

Agreed. I think a knob is appropriate, it's just a question of how that control exposed. Same answer to your question below.

-Alex

Quoted 17 lines
Quoted 11 lines

The hardware is capable of aggregating QMAP packets arriving on a connection into a single buffer, so this provides a way of requesting it do that.

Quoted 3 lines
Quoted 1 line

#define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1)

Similar here? If you have flow control you probably want to use it?

I agree with that, though perhaps there are cases where it is pointless, or can't be supported, so one might want to simply *not* implement/advertise the feature. I don't know.

Sure, but then that's likely something the driver would need to know, not necessarily userspace?

johannes

author
#99 lore raw

WWAN Controller Framework (was IPA [PATCH v2 00/17])

In reply to Alex Elder

OK I want to try to organize a little more concisely some of the discussion on this, because there is a very large amount of volume to date and I think we need to try to narrow the focus back down again.

I'm going to use a few terms here. Some of these I really don't like, but I want to be unambiguous *and* (at least for now) I want to avoid the very overloaded term "device".

I have lots more to say, but let's start with a top-level picture, to make sure we're all on the same page.

         WWAN Communication
         Channel (Physical)
                 |     ------------------------
------------     v     |           :+ Control |  \
|          |-----------|           :+ Data    |  |
|    AP    |           | WWAN unit :+ Voice   |   > Functions
|          |===========|           :+ GPS     |  |
------------     ^     |           :+ ...     |  /
                 |     -------------------------

Multiplexed WWAN
Communication
Channel (Physical)

- The *AP* is the main CPU complex that's running Linux on one or more CPU cores.
- A *WWAN unit* is an entity that shares one or more physical *WWAN communication channels* with the AP.
- A *WWAN communication channel* is a bidirectional means of carrying data between the AP and WWAN unit.
- A WWAN communication channel carries data using a *WWAN protocol*.
- A WWAN unit implements one or more *WWAN functions*, such as 5G data, LTE voice, GPS, and so on.
- A WWAN unit shall implement a *WWAN control function*, used to manage the use of other WWAN functions, as well as the WWAN unit itself.
- The AP communicates with a WWAN function using a WWAN protocol.
- A WWAN physical channel can be *multiplexed*, in which case it carries the data for one or more *WWAN logical channels*.
- A multiplexed WWAN communication channel uses a *WWAN wultiplexing protocol*, which is used to separate independent data streams carrying other WWAN protocols.
- A WWAN logical channel carries a bidirectional stream of WWAN protocol data between an entity on the AP and a WWAN function.

Does that adequately represent a very high-level picture of what we're trying to manage?

And if I understand it right, the purpose of the generic framework being discussed is to define a common mechanism for managing (i.e., discovering, creating, destroying, querying, configuring, enabling, disabling, etc.) WWAN units and the functions they implement, along with the communication and logical channels used to communicate with them.

Comments?

-Alex

#66 lore raw
In reply to Alex Elder
Quoted 19 lines · Alex Elder
Quoted 8 lines · Johannes Berg

Really there are two possible ways (and they intersect to some extent).

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

I *think* this model makes the most sense. But at this point it would take very little to convince me otherwise... (And then I saw Arnd's message advocating the other one, unfortunately...)

Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

Quoted 5 lines

What I don't like about this is that it's more monolithic. It seems better to have the low-level IPA or Intel modem driver (or any other driver that can support communication between the AP and WWAN device) present communication paths that other function- specific drivers can attach to and use.

I did not understand Johannes description as two competing models for the same code, but rather two kinds of existing hardware that a new driver system would have to deal with.

I was trying to simplify it to just having the second model, by adding a hack to support the first, but my view was rather unpopular so far, so if everyone agrees on one way to do it, don't worry about me ;-)

Quoted 6 lines
Quoted 3 lines

Now, it's not clear to me where IPA actually falls, because so far we've been talking about the IPA driver only as providing *netdevs*, not any control channels, so I'm not actually sure where the control channel is.

There is user space code that handles all of this, and as far as I can tell, parts of it will always remain proprietary.

Two replies on this:

- to answer Johannes question, my understanding is that the interface between kernel and firmware/hardware for IPA has a single 'struct device' that is used for both the data and the control channels, rather than having a data channel and an independent control device, so this falls into the same category as the Intel one (please correct me on that)

- The user space being proprietary is exactly what we need to avoid with the wwan subsystem. We need to be able to use the same method for setting up Intel, Qualcomm, Samsung, Unisoc or Hisilicon modems or anything else that hooks into the subsystem, and support that in network manager as well as the Android equivalent.
If Qualcomm wants to provide their own proprietary user space solution, we can't stop them, but then that should also work on all the others unless they intentionally break it. ;-)

Quoted 7 lines
Quoted 3 lines

and simply require that the channel is attached to the wwan device with the representation-specific call (wwan_attach_netdev, wwan_attach_tty, ...).

Or maybe have the WWAN device present interfaces with attributes, and have drivers that are appropriate for each interface attach to only the ones they recognize they support.

I think you both mean the same thing here, a structure with callback pointers that may or may not be filled by the driver depending on its capabilities.

What we should try to avoid though is a way to add driver private interfaces that risk having multiple drivers create similar functionality in incompatible ways.

Arnd

author
#100 lore raw

Re: WWAN Controller Framework (was IPA [PATCH v2 00/17])

In reply to Alex Elder

Sorry, I neglected to add Dan and Johannes--who have been primary contributors in this discussion--to this. Adding now.

-Alex

Quoted 60 lines · Alex Elder

OK I want to try to organize a little more concisely some of the discussion on this, because there is a very large amount of volume to date and I think we need to try to narrow the focus back down again.

I'm going to use a few terms here. Some of these I really don't like, but I want to be unambiguous *and* (at least for now) I want to avoid the very overloaded term "device".

I have lots more to say, but let's start with a top-level picture, to make sure we're all on the same page.

         WWAN Communication
         Channel (Physical)
                 |     ------------------------
------------     v     |           :+ Control |  \
|          |-----------|           :+ Data    |  |
|    AP    |           | WWAN unit :+ Voice   |   > Functions
|          |===========|           :+ GPS     |  |
------------     ^     |           :+ ...     |  /
                 |     -------------------------

Multiplexed WWAN
Communication
Channel (Physical)

- The *AP* is the main CPU complex that's running Linux on one or more CPU cores.
- A *WWAN unit* is an entity that shares one or more physical *WWAN communication channels* with the AP.
- A *WWAN communication channel* is a bidirectional means of carrying data between the AP and WWAN unit.
- A WWAN communication channel carries data using a *WWAN protocol*.
- A WWAN unit implements one or more *WWAN functions*, such as 5G data, LTE voice, GPS, and so on.
- A WWAN unit shall implement a *WWAN control function*, used to manage the use of other WWAN functions, as well as the WWAN unit itself.
- The AP communicates with a WWAN function using a WWAN protocol.
- A WWAN physical channel can be *multiplexed*, in which case it carries the data for one or more *WWAN logical channels*.
- A multiplexed WWAN communication channel uses a *WWAN wultiplexing protocol*, which is used to separate independent data streams carrying other WWAN protocols.
- A WWAN logical channel carries a bidirectional stream of WWAN protocol data between an entity on the AP and a WWAN function.

Does that adequately represent a very high-level picture of what we're trying to manage?

And if I understand it right, the purpose of the generic framework being discussed is to define a common mechanism for managing (i.e., discovering, creating, destroying, querying, configuring, enabling, disabling, etc.) WWAN units and the functions they implement, along with the communication and logical channels used to communicate with them.

Comments?

-Alex

#104 lore raw

Re: WWAN Controller Framework (was IPA [PATCH v2 00/17])

In reply to Alex Elder
Quoted 21 lines · Alex Elder

OK I want to try to organize a little more concisely some of the discussion on this, because there is a very large amount of volume to date and I think we need to try to narrow the focus back down again.

I'm going to use a few terms here. Some of these I really don't like, but I want to be unambiguous *and* (at least for now) I want to avoid the very overloaded term "device".

I have lots more to say, but let's start with a top-level picture, to make sure we're all on the same page.

         WWAN Communication
         Channel (Physical)
                 |     ------------------------
------------     v     |           :+ Control |  \
>          |-----------|           :+ Data    |  |
>    AP    |           | WWAN unit :+ Voice   |   > Functions
>          |===========|           :+ GPS     |  |
------------     ^     |           :+ ...     |  /
                 |     -------------------------
Quoted 8 lines

Multiplexed WWAN
Communication
Channel (Physical)

- The *AP* is the main CPU complex that's running Linux on one or more CPU cores.
- A *WWAN unit* is an entity that shares one or more physical *WWAN communication channels* with the AP.

You could just say "WWAN modem" here.

Quoted 5 lines

- A *WWAN communication channel* is a bidirectional means of carrying data between the AP and WWAN unit.
- A WWAN communication channel carries data using a *WWAN protocol*.
- A WWAN unit implements one or more *WWAN functions*, such as 5G data, LTE voice, GPS, and so on.

Go more generic here. Not just 5G data but any WWAN IP-based data (GPRS, EDGE, CDMA, UMTS, EVDO, LTE, 5G, etc). And not just LTE voice but any voice data; plenty of devices don't support LTE but still have "WWAN logical communication channels"

Quoted 6 lines

- A WWAN unit shall implement a *WWAN control function*, used to manage the use of other WWAN functions, as well as the WWAN unit itself.
- The AP communicates with a WWAN function using a WWAN protocol.
- A WWAN physical channel can be *multiplexed*, in which case it carries the data for one or more *WWAN logical channels*.

It's unclear to me what "physical" means here. USB Interface or Endpoint or PCI Function or SMD channel? Or kernel TTY device?

For example on Qualcomm-based USB dongles a given USB Interface's Endpoint represents a QMAP "IP data" channel which itself could be multiplexed into separate "IP data" channels. Or that USB Endpoint(s) could be exposed as a TTY which itself can be MUX-ed dynamically using GSM 07.10.

To me "physical" usually means the bus type (PCI, USB, SMD, whatever). A Linux hardware driver (IPA, qmi_wwan, option, sierra, etc) binds to that physical entity using hardware IDs (USB or PCI VID/PID, devicetree properties) and exposes some "WWAN logical communication channels". Those logical channels might be multiplexed and another driver (rmnet) could handle exposing the de-muxed logical channels that the muxed logical channel carries.

Quoted 5 lines

- A multiplexed WWAN communication channel uses a *WWAN wultiplexing protocol*, which is used to separate independent data streams carrying other WWAN protocols.
- A WWAN logical channel carries a bidirectional stream of WWAN protocol data between an entity on the AP and a WWAN function.

It *usually* is bidirectional. For example some GPS logical communication channels just start spitting out NMEA when you give the control function a command. The NMEA ports themselves don't accept any input.

Quoted 2 lines

Does that adequately represent a very high-level picture of what we're trying to manage?

Yes, pretty well. Thanks for trying to specify it all.

Quoted 6 lines

And if I understand it right, the purpose of the generic framework being discussed is to define a common mechanism for managing (i.e., discovering, creating, destroying, querying, configuring, enabling, disabling, etc.) WWAN units and the functions they implement, along with the communication and logical channels used to communicate with them.

Yes.

Dan

Quoted 3 lines

Comments?

-Alex

author
#105 lore raw

Re: WWAN Controller Framework (was IPA [PATCH v2 00/17])

Quoted 26 lines · Dan Williams
Quoted 25 lines · Alex Elder

OK I want to try to organize a little more concisely some of the discussion on this, because there is a very large amount of volume to date and I think we need to try to narrow the focus back down again.

I'm going to use a few terms here. Some of these I really don't like, but I want to be unambiguous *and* (at least for now) I want to avoid the very overloaded term "device".

I have lots more to say, but let's start with a top-level picture, to make sure we're all on the same page.

         WWAN Communication
         Channel (Physical)
                 |     ------------------------
------------     v     |           :+ Control |  \
>          |-----------|           :+ Data    |  |
>    AP    |           | WWAN unit :+ Voice   |   > Functions
>          |===========|           :+ GPS     |  |
------------     ^     |           :+ ...     |  /
                 |     -------------------------

Multiplexed WWAN
Communication
Channel (Physical)

Quoted 6 lines
Quoted 4 lines

- The *AP* is the main CPU complex that's running Linux on one or more CPU cores.
- A *WWAN unit* is an entity that shares one or more physical *WWAN communication channels* with the AP.

You could just say "WWAN modem" here.

That sounds great to me.

Quoted 10 lines
Quoted 5 lines

- A *WWAN communication channel* is a bidirectional means of carrying data between the AP and WWAN unit.
- A WWAN communication channel carries data using a *WWAN protocol*.
- A WWAN unit implements one or more *WWAN functions*, such as 5G data, LTE voice, GPS, and so on.

Go more generic here. Not just 5G data but any WWAN IP-based data (GPRS, EDGE, CDMA, UMTS, EVDO, LTE, 5G, etc). And not just LTE voice but any voice data; plenty of devices don't support LTE but still have "WWAN logical communication channels"

I really meant *any* sort of function, and was only trying to give a few examples. So yes, my meaning was completely generic, as you suggest.

Quoted 9 lines
Quoted 6 lines

- A WWAN unit shall implement a *WWAN control function*, used to manage the use of other WWAN functions, as well as the WWAN unit itself.
- The AP communicates with a WWAN function using a WWAN protocol.
- A WWAN physical channel can be *multiplexed*, in which case it carries the data for one or more *WWAN logical channels*.

It's unclear to me what "physical" means here. USB Interface or Endpoint or PCI Function or SMD channel? Or kernel TTY device?

I'm trying to distinguish between (let's say) "hardware" communication channels (such as what IPA basically provides) and logical ones, whose data is multiplexed over a "hardware"/"physical" channel. Maybe "link" would be a better term for what I referred to as a physical channel, and then have "channels" be multiplexed over a link. If you have a good suggestion, please offer it.

But I think yes, USB interface, TTY device, etc. are what I mean. I wanted to capture the fact that there might be more than one of these (for example a user space QMI link for control, and IPA link for data?), and that any one of these might also be multiplexed.

Multiplexing is a pretty basic capability of a network link, and I think the only reason I called it out separately is to be explicit that it is needs to be supported.

Quoted 5 lines

For example on Qualcomm-based USB dongles a given USB Interface's Endpoint represents a QMAP "IP data" channel which itself could be multiplexed into separate "IP data" channels. Or that USB Endpoint(s) could be exposed as a TTY which itself can be MUX-ed dynamically using GSM 07.10.

Yeah. In this case the hardware connection between the AP and the USB dongle would provide a WWAN link (which I think corresponds to the QMAP "IP data" channel). And if you wanted to multiplex that you would use a multiplexing protocol (like QMAP). And that protocol would carry one or more logical channels, each using its own WWAN protocol. It sounds like GSM 07.10 would be another WWAN multiplexing protocol.

Quoted 4 lines

To me "physical" usually means the bus type (PCI, USB, SMD, whatever). A Linux hardware driver (IPA, qmi_wwan, option, sierra, etc) binds to that physical entity using hardware IDs (USB or PCI VID/PID, devicetree properties) and exposes some "WWAN logical communication channels".

(Or perhaps exposes the ability to create "WWAN logical channels.")

"Physical" is probably not a good term. And perhaps focusing on the transport isn't right--maybe the focus should mainly be on the WWAN modem entity. But one of the things we're trying to address here is that there might be several distinct "physical" paths through which the AP and modem can communicate, so a driver's ability to provide such a path should be a sort of first class abstraction.

I'm really trying to get this discussion to converge a little, to have a few anchor points to build on. I hope we're getting there.

Quoted 10 lines

Those logical channels might be multiplexed and another driver (rmnet) could handle exposing the de-muxed logical channels that the muxed logical channel carries.

Quoted 5 lines

- A multiplexed WWAN communication channel uses a *WWAN wultiplexing protocol*, which is used to separate independent data streams carrying other WWAN protocols.
- A WWAN logical channel carries a bidirectional stream of WWAN protocol data between an entity on the AP and a WWAN function.

Quoted 4 lines

It *usually* is bidirectional. For example some GPS logical communication channels just start spitting out NMEA when you give the control function a command. The NMEA ports themselves don't accept any input.

That's fine... I don't think there's anything wrong with a particular application not using both directions.

Quoted 4 lines
Quoted 2 lines

Does that adequately represent a very high-level picture of what we're trying to manage?

Yes, pretty well. Thanks for trying to specify it all.

I think we're making progress. I have some more thoughts but I think I'll wait until tomorrow to share them.

Thanks a lot Dan. At some point it might be better to have a conference call to make better progress, but I'm not suggesting that yet.

-Alex

Quoted 15 lines
Quoted 6 lines

And if I understand it right, the purpose of the generic framework being discussed is to define a common mechanism for managing (i.e., discovering, creating, destroying, querying, configuring, enabling, disabling, etc.) WWAN units and the functions they implement, along with the communication and logical channels used to communicate with them.

Yes.

Dan

Quoted 3 lines

Comments?

-Alex

reviewer
#52 lore raw
In reply to Alex Elder

Hi Alex,

I'll just pick a few or your messages and reply there - some other subthreads seem to have pretty much completed.

Quoted 11 lines

Sorry for the delay. There's a lot here to go through, and with each message the picture is (slowly) getting a bit clearer for me. Still, there are some broad tradeoffs to consider and I think we need to get a little more specific again. I'm going to start a new thread (or rather re-subject a response to the very first one) that tries to do a fresh start that takes into account the discussion so far.

I will also be talking with some people inside Qualcomm (including Subash) soon to make sure we don't miss any requirements or insights they know of that I don't realize are important.

That's much appreciated.

Quoted 8 lines
Quoted 4 lines

Linux usually tries to keep drivers generic and focused; each driver is written for a specific function. For example, a USB device usually provides multiple USB interfaces which will be bound to different Linux drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc.

So USB has some attributes similar to what we're talking about here. But if I'm not mistaken we want some sort of an overall management scheme as well.

Yes. For the record, I think the part about "keep drivers generic and focused" really only works for USB devices that expose different pieces that look like any other network device or a TTY device on the USB level, just the combination of these things (and knowing about that) really makes them a modem.

For things like IPA or the (hypothetical) Intel driver we're talking about, it's still all managed by a single (PCIe) driver. For the Intel device in particular, even all the control channels are over exactly the same transport mechanism as the data channels.

Quoted 7 lines
Quoted 5 lines

These drivers are often generic and we may not have enough information in one driver to know that the parent of this interface is a WWAN device. But another driver might. Since probing is asynchronous we may have cdc-acm bind to a device and provide a TTY before cdc-ether (which does know it's a WWAN) binds and provides the netdevice.

Is this why Johannes wanted to have a "maybe attach" method?

Yes.

Quoted 9 lines

I don't like the "maybe" API unless there's no other way to do it.

Instead I think it would be better for the probing driver to register with a whatever the WWAN core is, and then have the WWAN core be responsible for pulling things all together when it receives a request to do so. I.e., something in user space should request that a registered data interface be brought up, and at that time everything "knows" it's implemented as part of a WWAN device.

Right, but then we just punt to userspace. Mostly we *do* (eventually!) know that it's a WWAN device, just not every component can detect it. Some components typically can.

So for example, you might have a USB multi-function device with a network function (looks just like ethernet pretty much) but another TTY control channel that actually has some specific WWAN IDs, so that part can know it's a WWAN.

Here, the ethernet function would need "maybe" attach, and the control channel would "definitively" attach, pulling it together as a WWAN device without requiring any more action or information.

Quoted 8 lines

So maybe:
- Hardware probe detects a WWAN device
- The drivers that detect the WWAN device register it with the WWAN core code.
- A control channel is instantiated at/before the time the WWAN device is registered
- Something in user space should manage the bring-up of any other things on the WWAN device thereafter

But those things need to actually get connected first :-)

In IPA/Intel case this is easy since it's a single driver. But if there's multi-function device with ethernet being a completely separate driver, the control channel cannot even reach that to tell it to create a new data channel.

Quoted 5 lines
Quoted 3 lines

userspace should probably always create the netdevices (since they are always useless until userspace coordinates with the firmware about them) but that's not how things are yet.

That's too bad. How hard would that be to change?

Depends, but as I said above it's probably orthogonal to the question. The data channel driver would still need to attach to the WWAN device somehow so it becomes reachable by the control plane (note this isn't the same as "control channel" since the latter talks to the modem, the control plane talks to the kernel drivers).

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

- What causes a created channel to be removed?

Driver removal, userspace WWAN daemon terminating the packet data connection which the channel represents, the modem terminating the packet data connection (eg network initiated disconnect), etc.

OK this is as I expected. Driver (or device) removal is somewhat obvious, but you're confirming user space might request it as well.

If userspace actually had the ability to create (data) channels, then it would have the ability to also remove them. Right now, this may or may not be supported by the drivers that act together to form the interfaces to a WWAN device.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

- You distinguish between attaching a netdevice and (what I'll call) activating it. What causes activation?

Can you describe what you mean by "activating"? Do you mean successfully TX/RX packets via the netdev and the outside world?

Quoted 7 lines

Johannes mentioned an API to "maybe attach" a device. That begs the question of what happens if this request does *not* attach. Does the attach request have to be made again, or is it done automatically with a notification, or something else?

So by "activation" I was trying to refer to the notion of this subsequent successful attach.

Oh. Well, what I was thinking that "maybe attach" would just be a sort of "in-limbo" WWAN device that doesn't get visible to userspace or the control plane until something did a "definitively attach" to it so it was known to be a WWAN device.

The case of "maybe attach but never get to definitive attach" would be the case where the USB driver bound a real ethernet device, for example, not something that looks like an ethernet device but really is part of a modem.

OTOH, "activating" a data channel is also needed somehow through the control channel by talking to the modem, i.e. making a connection. In the ideal case we'd not even *have* a netdev until it makes sense to create a data channel, but in reality a lot of devices have one around all the time (or even only support one), AFAICT.

Quoted 8 lines
Quoted 3 lines

I read "attach" here as simply associating an existing netdev with the "parent" WWAN device. A purely Linux operation that is only book- keeping and may not have any interaction with the modem.

If that's the case I would want the "activation" to be a separate step. The attach would do the bookkeeping, and generally shouldn't fail. An attached interface would be brought up ("activated") separately and might fail if things aren't quite ready yet.

Right, but netdevs need to be brought up anyway, and that can fail?

Quoted 10 lines
Quoted 7 lines
Quoted 2 lines

- How are the attributes of a WWAN device or channel set, or communicated?

Via netlink attributes when userspace asks the WWAN device to create a new channel. In the control methods I've seen, only userspace really knows the channel identifier that it and the modem have agreed on (eg what the MUX ID in the QMAP header would be, or the MBIM Session ID).

Yes, that's the way it's worked for rmnet and IPA. Previously it was IOCTL requests but it's currently hard-wired.

Right. We're just trying to lift it out of the Qualcomm sphere into something more generically useful.

johannes

reviewer
#67 lore raw
In reply to Arnd Bergmann
Quoted 26 lines · Arnd Bergmann
Quoted 19 lines · Alex Elder
Quoted 8 lines · Johannes Berg

Really there are two possible ways (and they intersect to some extent).

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

I *think* this model makes the most sense. But at this point it would take very little to convince me otherwise... (And then I saw Arnd's message advocating the other one, unfortunately...)

Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

Quoted 5 lines

What I don't like about this is that it's more monolithic. It seems better to have the low-level IPA or Intel modem driver (or any other driver that can support communication between the AP and WWAN device) present communication paths that other function- specific drivers can attach to and use.

Quoted 3 lines

I did not understand Johannes description as two competing models for the same code, but rather two kinds of existing hardware that a new driver system would have to deal with.

Right.

Quoted 3 lines

I was trying to simplify it to just having the second model, by adding a hack to support the first, but my view was rather unpopular so far, so if everyone agrees on one way to do it, don't worry about me ;-)

:-)

However, to also reply to Alex: I don't know exactly how IPA works, but for the Intel modem at least you can't fundamentally have two drivers for different parts of the functionality, since it's just a single piece of hardware and you need to allocate hardware resources from a common pool etc. So you cannot split the driver into "Intel modem control channel driver" and "Intel modem data channel driver". In fact, it's just a single "struct device" on the PCIe bus that you can bind to, and only one driver can bind at a time.

So, IOW, I'm not sure I see how you'd split that up. I guess you could if you actually do something like the "rmnet" model, and I suppose you're free to do that for IPA if you like, but I tend to think that's actually a burden, not a win since you just get more complex code that needs to interact with more pieces. A single driver for a single hardware that knows about the few types of channels seems simpler to me.

Quoted 6 lines

- to answer Johannes question, my understanding is that the interface between kernel and firmware/hardware for IPA has a single 'struct device' that is used for both the data and the control channels, rather than having a data channel and an independent control device, so this falls into the same category as the Intel one (please correct me on that)

That sounds about the same then, right.

Are the control channels to IPA are actually also tunnelled over the rmnet protocol? And even if they are, perhaps they have a different hardware queue or so? That'd be the case for Intel - different hardware queue, same (or at least similar) protocol spoken for the DMA hardware itself, but different contents of the messages obviously.

Quoted 9 lines

- The user space being proprietary is exactly what we need to avoid with the wwan subsystem. We need to be able to use the same method for setting up Intel, Qualcomm, Samsung, Unisoc or Hisilicon modems or anything else that hooks into the subsystem, and support that in network manager as well as the Android equivalent.
If Qualcomm wants to provide their own proprietary user space solution, we can't stop them, but then that should also work on all the others unless they intentionally break it. ;-)

:-)

I tend to think there's always going to be some level of specific handling here, because e.g. the Intel modem can expose MBIM or AT command control channels, but not much else (that'd be useful for us anyway, since we don't know how to speak debug protocol etc.). Other modems will expose *only* AT commands, or *only* MBIM, and yet others may also offer QMI and then that might be preferable.

Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

and simply require that the channel is attached to the wwan device with the representation-specific call (wwan_attach_netdev, wwan_attach_tty, ...).

Or maybe have the WWAN device present interfaces with attributes, and have drivers that are appropriate for each interface attach to only the ones they recognize they support.

I think you both mean the same thing here, a structure with callback pointers that may or may not be filled by the driver depending on its capabilities.

:-)

Quoted 3 lines

What we should try to avoid though is a way to add driver private interfaces that risk having multiple drivers create similar functionality in incompatible ways.

Right.

johannes

reviewer
#101 lore raw

Re: WWAN Controller Framework (was IPA [PATCH v2 00/17])

In reply to Alex Elder
Quoted 4 lines · Alex Elder
Quoted 4 lines

OK I want to try to organize a little more concisely some of the discussion on this, because there is a very large amount of volume to date and I think we need to try to narrow the focus back down again.

Sounds good to me!

Quoted 15 lines
Quoted 15 lines

I'm going to use a few terms here. Some of these I really don't like, but I want to be unambiguous *and* (at least for now) I want to avoid the very overloaded term "device".

I have lots more to say, but let's start with a top-level picture, to make sure we're all on the same page.

         WWAN Communication
         Channel (Physical)
                 |     ------------------------
------------     v     |           :+ Control |  \
>          |-----------|           :+ Data    |  |
>    AP    |           | WWAN unit :+ Voice   |   > Functions
>          |===========|           :+ GPS     |  |
Quoted 5 lines
Quoted 5 lines
------------     ^     |           :+ ...     |  /
                 |     -------------------------

Multiplexed WWAN
Communication
Channel (Physical)

Sounds right to me. I'm not sure if you're distinguishing here between the "Data function" and multiple data channels to the data function, but at this point I guess it doesn't matter.

Quoted 4 lines
Quoted 4 lines

- The *AP* is the main CPU complex that's running Linux on one or more CPU cores.
- A *WWAN unit* is an entity that shares one or more physical *WWAN communication channels* with the AP.

Quoted 8 lines
Quoted 8 lines

- A *WWAN communication channel* is a bidirectional means of carrying data between the AP and WWAN unit.
- A WWAN communication channel carries data using a *WWAN protocol*.
- A WWAN unit implements one or more *WWAN functions*, such as 5G data, LTE voice, GPS, and so on.
- A WWAN unit shall implement a *WWAN control function*, used to manage the use of other WWAN functions, as well as the WWAN unit itself.

I think here we need to be more careful. I don't know how you want to call it, but we actually have multiple levels of control here.

You have
* hardware control, to control how you actually use the (multiple or not) physical communication channel(s) to the WWAN unit
* this is partially exposed to userspace via the WWAN netlink family or something like that, so userspace can create new netdevs to tx/rx with the "data function" and to the network; note that it could be one or multiple
* WWAN control, which is typically userspace communicating with the WWAN control function in the WWAN unit, but this can take different forms (as I mentioned earlier, e.g. AT commands, MBIM, QMI)

Quoted 1 line
Quoted 1 line

- The AP communicates with a WWAN function using a WWAN protocol.

Right, that's just device specific (IPA vs. Intel vs. ...)

Quoted 2 lines
Quoted 2 lines

- A WWAN physical channel can be *multiplexed*, in which case it carries the data for one or more *WWAN logical channels*.

This ... depends a bit on how you exactly define a physical channel here. Is that, to you, the PCIe/USB link? In that case, yes, obviously you have only one physical channel for each WWAN unit.

However, I'd probably see this slightly differently, because e.g. the Intel modem has multiple DMA engines, and so you actually have multiple DMA rings to talk to the WWAN unit, and I'd have called each DMA ring a physical channel. And then, you just have a 1:1 from physical to logical channel since it doesn't actually carry a multiplexing protocol.

Quoted 3 lines
Quoted 3 lines

- A multiplexed WWAN communication channel uses a *WWAN wultiplexing protocol*, which is used to separate independent data streams carrying other WWAN protocols.

Like just described, this isn't really needed and is a device-specific property.

Quoted 5 lines
Quoted 5 lines

- A WWAN logical channel carries a bidirectional stream of WWAN protocol data between an entity on the AP and a WWAN function.

Does that adequately represent a very high-level picture of what we're trying to manage?

Pretty much.

I only disagree slightly on the control planes (there are multiple, and multiple options for the "Control function" one), and on the whole notion of physical link/logical link/multiplexing which is device specific.

Quoted 6 lines
Quoted 6 lines

And if I understand it right, the purpose of the generic framework being discussed is to define a common mechanism for managing (i.e., discovering, creating, destroying, querying, configuring, enabling, disabling, etc.) WWAN units and the functions they implement, along with the communication and logical channels used to communicate with them.

Well, some subset of that matrix, the framework won't actually destroy WWAN units I hope ;-)

But yes. I'd probably captured it in layers, and say that we have a

WWAN framework layer
- discover, query, configure WWAN units
- enable, disable channels to the functions inside the WWAN units

WWAN device driver
- implement (partial) API offered by WWAN framework layer to allow these things
(sometimes may not allow creating more control or data channels for example, and fixed function channels are precreated, but then can still discover data about the device and configure the channels
- implement the device-specific protocols etc. necessary to achieve this

Userspace
- uses control function channel (e.g. TTY) to talk directly to the WWAN unit's control function
- uses WWAN framework APIs to create/configure/... (other) function channels
(it may be necessary to create a control channel even, before being able to use it, since different options (AT/MBIM/QMI) may be there
- configures netdevs (data function channels) after their creation

johannes

author
#53 lore raw
In reply to Johannes Berg
Quoted 5 lines · Johannes Berg

Hi Alex,

I'll just pick a few or your messages and reply there - some other subthreads seem to have pretty much completed.

. . .

Quoted 15 lines
Quoted 8 lines
Quoted 4 lines

Linux usually tries to keep drivers generic and focused; each driver is written for a specific function. For example, a USB device usually provides multiple USB interfaces which will be bound to different Linux drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc.

So USB has some attributes similar to what we're talking about here. But if I'm not mistaken we want some sort of an overall management scheme as well.

Yes. For the record, I think the part about "keep drivers generic and focused" really only works for USB devices that expose different pieces that look like any other network device or a TTY device on the USB level, just the combination of these things (and knowing about that) really makes them a modem.

Quoted 4 lines

For things like IPA or the (hypothetical) Intel driver we're talking about, it's still all managed by a single (PCIe) driver. For the Intel device in particular, even all the control channels are over exactly the same transport mechanism as the data channels.

Actually the setup for IPA requires certain things to be done via QMI by something outside the IPA driver, and it uses a separate communication path. But I understand what you're saying.

. . .

Quoted 10 lines
Quoted 9 lines

I don't like the "maybe" API unless there's no other way to do it.

Instead I think it would be better for the probing driver to register with a whatever the WWAN core is, and then have the WWAN core be responsible for pulling things all together when it receives a request to do so. I.e., something in user space should request that a registered data interface be brought up, and at that time everything "knows" it's implemented as part of a WWAN device.

Quoted 3 lines

Right, but then we just punt to userspace. Mostly we *do* (eventually!) know that it's a WWAN device, just not every component can detect it. Some components typically can.

We need to identify the existence of a WWAN device (which is I guess--typically? always?--a modem). Perhaps that can be discovered in some cases but I think it means a node described by Device Tree.

Quoted 8 lines

So for example, you might have a USB multi-function device with a network function (looks just like ethernet pretty much) but another TTY control channel that actually has some specific WWAN IDs, so that part can know it's a WWAN.

Here, the ethernet function would need "maybe" attach, and the control channel would "definitively" attach, pulling it together as a WWAN device without requiring any more action or information.

So you're saying you have a single Ethernet driver, and it can drive an Ethernet device connected to a WWAN, or not connected to a WWAN, without any changes. The only distinction is that if the device is part of a WWAN it needs to register with the WWAN framework. Is that right?

Quoted 10 lines
Quoted 8 lines

So maybe:
- Hardware probe detects a WWAN device
- The drivers that detect the WWAN device register it with the WWAN core code.
- A control channel is instantiated at/before the time the WWAN device is registered
- Something in user space should manage the bring-up of any other things on the WWAN device thereafter

But those things need to actually get connected first :-)

What I meant is that the registering with the "WWAN core code" is what does that "connecting." The WWAN code has the information about what got registered. But as I said above, this WWAN device needs to be identified, and I think (at least for IPA) that will require something in Device Tree. That will "connect" them.

Or I might be misunderstanding your point.

Quoted 4 lines

In IPA/Intel case this is easy since it's a single driver. But if there's multi-function device with ethernet being a completely separate driver, the control channel cannot even reach that to tell it to create a new data channel.

There's a lot more to talk about with control. I think you discuss this in another message, and I'll get to that shortly. But I think understand your point, and again I think it comes back to having an abstraction that represents the modem, distinct from (but "connected" to) the functions it implements/includes.

Quoted 18 lines
Quoted 5 lines
Quoted 3 lines

userspace should probably always create the netdevices (since they are always useless until userspace coordinates with the firmware about them) but that's not how things are yet.

That's too bad. How hard would that be to change?

Depends, but as I said above it's probably orthogonal to the question. The data channel driver would still need to attach to the WWAN device somehow so it becomes reachable by the control plane (note this isn't the same as "control channel" since the latter talks to the modem, the control plane talks to the kernel drivers).

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

- What causes a created channel to be removed?

Driver removal, userspace WWAN daemon terminating the packet data connection which the channel represents, the modem terminating the packet data connection (eg network initiated disconnect), etc.

Quoted 7 lines
Quoted 2 lines

OK this is as I expected. Driver (or device) removal is somewhat obvious, but you're confirming user space might request it as well.

If userspace actually had the ability to create (data) channels, then it would have the ability to also remove them. Right now, this may or may not be supported by the drivers that act together to form the interfaces to a WWAN device.

I think this (user space control) needs to be an option, but it doesn't have to be the only way.

. . .

You made some other good clarifications in this message but I'm going to try to capture them elsewhere rather than respond here.

-Alex

author
#68 lore raw
In reply to Johannes Berg
Quoted 39 lines · Johannes Berg
Quoted 26 lines · Arnd Bergmann
Quoted 19 lines · Alex Elder
Quoted 8 lines · Johannes Berg

Really there are two possible ways (and they intersect to some extent).

One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together.

I *think* this model makes the most sense. But at this point it would take very little to convince me otherwise... (And then I saw Arnd's message advocating the other one, unfortunately...)

Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

Quoted 5 lines

What I don't like about this is that it's more monolithic. It seems better to have the low-level IPA or Intel modem driver (or any other driver that can support communication between the AP and WWAN device) present communication paths that other function- specific drivers can attach to and use.

Quoted 3 lines

I did not understand Johannes description as two competing models for the same code, but rather two kinds of existing hardware that a new driver system would have to deal with.

Right.

Quoted 3 lines

I was trying to simplify it to just having the second model, by adding a hack to support the first, but my view was rather unpopular so far, so if everyone agrees on one way to do it, don't worry about me ;-)

:-)

Quoted 8 lines

However, to also reply to Alex: I don't know exactly how IPA works, but for the Intel modem at least you can't fundamentally have two drivers for different parts of the functionality, since it's just a single piece of hardware and you need to allocate hardware resources from a common pool etc. So you cannot split the driver into "Intel modem control channel driver" and "Intel modem data channel driver". In fact, it's just a single "struct device" on the PCIe bus that you can bind to, and only one driver can bind at a time.

Interesting. So a single modem driver needs to implement *all* of the features/functions? Like GPS or data log or whatever, all needs to share the same struct device? Or does what you're describing apply to a subset of the modem's functionality? Or something else?

Quoted 13 lines

So, IOW, I'm not sure I see how you'd split that up. I guess you could if you actually do something like the "rmnet" model, and I suppose you're free to do that for IPA if you like, but I tend to think that's actually a burden, not a win since you just get more complex code that needs to interact with more pieces. A single driver for a single hardware that knows about the few types of channels seems simpler to me.

Quoted 6 lines

- to answer Johannes question, my understanding is that the interface between kernel and firmware/hardware for IPA has a single 'struct device' that is used for both the data and the control channels, rather than having a data channel and an independent control device, so this falls into the same category as the Intel one (please correct me on that)

I don't think that's quite right, but it might be partially right. There is a single device representing IPA, but the picture is a little more complicated.

The IPA hardware is actually something that sits *between* the AP and the modem. It implements one form of communication pathway (IP data), but there are others (including QMI, which presents a network-like interface but it's actually implemented via clever use of shared memory and interrupts).

What we're talking about here is WWAN/modem management more generally though. It *sounds* like the Intel modem is more like a single device, which requires a single driver, that seems to implement a bunch of distinct functions.

On this I'm not very knowledgeable but for Qualcomm there is user space code that is in charge of overall management of the modem. It implements what I think you're calling control functions, negotiating with the modem to allow new data channels to be created. Normally the IPA driver would provide information to user space about available resources, but would only make a communication pathway available when requested.

I'm going to leave it at that for now.

Quoted 7 lines

That sounds about the same then, right.

Are the control channels to IPA are actually also tunnelled over the rmnet protocol? And even if they are, perhaps they have a different hardware queue or so? That'd be the case for Intel - different hardware queue, same (or at least similar) protocol spoken for the DMA hardware itself, but different contents of the messages obviously.

I want to be careful talking about "control" but for IPA it comes from user space. For the purpose of getting initial code upstream, all of that control functionality (which was IOCTL based) has been removed, and a fixed configuration is assumed.

Quoted 9 lines
Quoted 9 lines

- The user space being proprietary is exactly what we need to avoid with the wwan subsystem. We need to be able to use the same method for setting up Intel, Qualcomm, Samsung, Unisoc or Hisilicon modems or anything else that hooks into the subsystem, and support that in network manager as well as the Android equivalent.
If Qualcomm wants to provide their own proprietary user space solution, we can't stop them, but then that should also work on all the others unless they intentionally break it. ;-)

I won't comment on this, in part because I really don't know right now what is proprietary or why. I think that having user space (proprietary or not) be able to provide management capability is a good thing. If a unified kernel interface provides a common/generic way to manage the modem, I don't know why Qualcomm wouldn't adapt their code to use it. But I can't really speak for Qualcomm.

. . .

-Alex

author
#102 lore raw

Re: WWAN Controller Framework (was IPA [PATCH v2 00/17])

In reply to Johannes Berg
Quoted 8 lines · Johannes Berg
Quoted 4 lines · Alex Elder
Quoted 4 lines

OK I want to try to organize a little more concisely some of the discussion on this, because there is a very large amount of volume to date and I think we need to try to narrow the focus back down again.

Sounds good to me!

. . .

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

- A WWAN unit shall implement a *WWAN control function*, used to manage the use of other WWAN functions, as well as the WWAN unit itself.

I think here we need to be more careful. I don't know how you want to call it, but we actually have multiple levels of control here.

I completely agree with you. From what I understand there exists a control channel (or even more than one?) that serves a very specific purpose in modem management. The main reason I mention the WWAN control function is that someone (maybe you) indicated that a control channel automatically gets created.

But I agree, we need to be careful to avoid confusion here.

Quoted 13 lines

You have
* hardware control, to control how you actually use the (multiple or not) physical communication channel(s) to the WWAN unit
* this is partially exposed to userspace via the WWAN netlink family or something like that, so userspace can create new netdevs to tx/rx with the "data function" and to the network; note that it could be one or multiple
* WWAN control, which is typically userspace communicating with the WWAN control function in the WWAN unit, but this can take different forms (as I mentioned earlier, e.g. AT commands, MBIM, QMI)

Quoted 1 line
Quoted 1 line

- The AP communicates with a WWAN function using a WWAN protocol.

Quoted 8 lines

Right, that's just device specific (IPA vs. Intel vs. ...)

Quoted 2 lines
Quoted 2 lines

- A WWAN physical channel can be *multiplexed*, in which case it carries the data for one or more *WWAN logical channels*.

This ... depends a bit on how you exactly define a physical channel here. Is that, to you, the PCIe/USB link? In that case, yes, obviously you have only one physical channel for each WWAN unit.

I think that was what I was trying to capture. There exists one or more "physical" communication paths between the AP and WWAN unit/modem. And while one path *could* carry just one type of traffic, it could also carry multiple logical channels of traffic by multiplexing.

Quoted 5 lines

However, I'd probably see this slightly differently, because e.g. the Intel modem has multiple DMA engines, and so you actually have multiple DMA rings to talk to the WWAN unit, and I'd have called each DMA ring a physical channel. And then, you just have a 1:1 from physical to logical channel since it doesn't actually carry a multiplexing protocol.

Understood.

. . .

Quoted 12 lines

I only disagree slightly on the control planes (there are multiple, and multiple options for the "Control function" one), and on the whole notion of physical link/logical link/multiplexing which is device specific.

Quoted 6 lines
Quoted 6 lines

And if I understand it right, the purpose of the generic framework being discussed is to define a common mechanism for managing (i.e., discovering, creating, destroying, querying, configuring, enabling, disabling, etc.) WWAN units and the functions they implement, along with the communication and logical channels used to communicate with them.

Quoted 2 lines

Well, some subset of that matrix, the framework won't actually destroy WWAN units I hope ;-)

Hardware self-destruct would be an optional behavior.

Quoted 15 lines

But yes. I'd probably captured it in layers, and say that we have a

WWAN framework layer
- discover, query, configure WWAN units
- enable, disable channels to the functions inside the WWAN units

WWAN device driver
- implement (partial) API offered by WWAN framework layer to allow these things
(sometimes may not allow creating more control or data channels for example, and fixed function channels are precreated, but then can still discover data about the device and configure the channels
- implement the device-specific protocols etc. necessary to achieve this

Quoted 8 lines

Userspace
- uses control function channel (e.g. TTY) to talk directly to the WWAN unit's control function
- uses WWAN framework APIs to create/configure/... (other) function channels
(it may be necessary to create a control channel even, before being able to use it, since different options (AT/MBIM/QMI) may be there
- configures netdevs (data function channels) after their creation

I don't think I have any argument with this. I'm going to try to put together something that goes beyond what I wrote in this message, to try to capture what I think we agree on in a sort of loose design document.

Thanks Johannes.

-Alex

author
#72 lore raw
In reply to Arnd Bergmann
Quoted 4 lines · Arnd Bergmann
Quoted 3 lines · Alex Elder
Quoted 1 line · Johannes Berg

Really there are two possible ways (and they intersect to some extent).

. . .

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels.

What I don't like about this is that it's more monolithic. It seems better to have the low-level IPA or Intel modem driver (or any other driver that can support communication between the AP and WWAN device) present communication paths that other function- specific drivers can attach to and use.

Quoted 3 lines

I did not understand Johannes description as two competing models for the same code, but rather two kinds of existing hardware that a new driver system would have to deal with.

Based on my understanding of what he said in a message I just responded to, I think you are exactly right.

. . .

Quoted 3 lines

What we should try to avoid though is a way to add driver private interfaces that risk having multiple drivers create similar functionality in incompatible ways.

Agreed.

-Alex

#69 lore raw
In reply to Alex Elder
Quoted 21 lines · Alex Elder
Quoted 15 lines · Johannes Berg

On Mon, 2019-06-24 at 18:40 +0200, Arnd Bergmann wrote:

So, IOW, I'm not sure I see how you'd split that up. I guess you could if you actually do something like the "rmnet" model, and I suppose you're free to do that for IPA if you like, but I tend to think that's actually a burden, not a win since you just get more complex code that needs to interact with more pieces. A single driver for a single hardware that knows about the few types of channels seems simpler to me.

Quoted 6 lines

- to answer Johannes question, my understanding is that the interface between kernel and firmware/hardware for IPA has a single 'struct device' that is used for both the data and the control channels, rather than having a data channel and an independent control device, so this falls into the same category as the Intel one (please correct me on that)

I don't think that's quite right, but it might be partially right. There is a single device representing IPA, but the picture is a little more complicated.

Quoted 5 lines

The IPA hardware is actually something that sits *between* the AP and the modem. It implements one form of communication pathway (IP data), but there are others (including QMI, which presents a network-like interface but it's actually implemented via clever use of shared memory and interrupts).

Can you clarify how QMI fits in here? Do you mean one has to talk to both IPA and QMI to use the modem, or are these two alternative implementations for the same basic purpose?

Quoted 8 lines
Quoted 7 lines

That sounds about the same then, right.

Are the control channels to IPA are actually also tunnelled over the rmnet protocol? And even if they are, perhaps they have a different hardware queue or so? That'd be the case for Intel - different hardware queue, same (or at least similar) protocol spoken for the DMA hardware itself, but different contents of the messages obviously.

Quoted 4 lines

I want to be careful talking about "control" but for IPA it comes from user space. For the purpose of getting initial code upstream, all of that control functionality (which was IOCTL based) has been removed, and a fixed configuration is assumed.

My previous understanding was that from the hardware perspective there is only one control interface, which is for IPA. Part of this is abstracted to user space with ioctl commands to the IPA driver, and then one must set up rmnet to match these by configuring an rmnet device over netlink messages from user space, but rmnet does not have a control protocol with the hardware.

The exception here is the flow control, which is handled using in-band XON/OFF messages from the modem to rmnet (and corresponding Acks the other way) that IPA just passes through.

If we also need to talk to QMI, that would be something completely different though.

Arnd

reviewer
#71 lore raw
In reply to Alex Elder
Quoted 9 lines · Alex Elder
Quoted 8 lines

However, to also reply to Alex: I don't know exactly how IPA works, but for the Intel modem at least you can't fundamentally have two drivers for different parts of the functionality, since it's just a single piece of hardware and you need to allocate hardware resources from a common pool etc. So you cannot split the driver into "Intel modem control channel driver" and "Intel modem data channel driver". In fact, it's just a single "struct device" on the PCIe bus that you can bind to, and only one driver can bind at a time.

Quoted 5 lines

Interesting. So a single modem driver needs to implement *all* of the features/functions? Like GPS or data log or whatever, all needs to share the same struct device? Or does what you're describing apply to a subset of the modem's functionality? Or something else?

Well, what is even the "implement the functions"? I mean, as kernel drivers we're really just in the business of providing communication channels to those functions. E.g. if you have a GNSS/GPS device, you might just have another TTY channel with NMEA data coming out, or something like that, right?

But from a kernel POV yes, I don't see how you could create multiple function drivers for something behind the same PCIe device (unless it actually appeared as multiple virtual functions or such, like the bigger ethernet NICs, but it doesn't).

But this points out to me that I was actually not quite accurate when I spoke about struct device before in the USB context with function drivers, but I have to do some research before I can correct myself correctly.

Quoted 18 lines
Quoted 13 lines

So, IOW, I'm not sure I see how you'd split that up. I guess you could if you actually do something like the "rmnet" model, and I suppose you're free to do that for IPA if you like, but I tend to think that's actually a burden, not a win since you just get more complex code that needs to interact with more pieces. A single driver for a single hardware that knows about the few types of channels seems simpler to me.

Quoted 6 lines

- to answer Johannes question, my understanding is that the interface between kernel and firmware/hardware for IPA has a single 'struct device' that is used for both the data and the control channels, rather than having a data channel and an independent control device, so this falls into the same category as the Intel one (please correct me on that)

I don't think that's quite right, but it might be partially right. There is a single device representing IPA, but the picture is a little more complicated.

Quoted 5 lines

The IPA hardware is actually something that sits *between* the AP and the modem. It implements one form of communication pathway (IP data), but there are others (including QMI, which presents a network-like interface but it's actually implemented via clever use of shared memory and interrupts).

OK.

Well, I guess this then might eventually become a bit of a hybrid - you eventually want one WWAN device to represent it all to userspace, but might actually have multiple devices with different drivers (from the kernel POV)?

But this is more like all the USB devices work. I just have to figure out how to correctly tie them together - "struct device" may or may not be right? I need to check how this functions.

I guess for something where you have DT (like you allude to elsewhere) you could just capture all this in DT by having some phandle link or something?

Quoted 4 lines

What we're talking about here is WWAN/modem management more generally though. It *sounds* like the Intel modem is more like a single device, which requires a single driver, that seems to implement a bunch of distinct functions.

Yes.

Quoted 7 lines

On this I'm not very knowledgeable but for Qualcomm there is user space code that is in charge of overall management of the modem. It implements what I think you're calling control functions, negotiating with the modem to allow new data channels to be created. Normally the IPA driver would provide information to user space about available resources, but would only make a communication pathway available when requested.

Right.

Quoted 10 lines
Quoted 5 lines

Are the control channels to IPA are actually also tunnelled over the rmnet protocol? And even if they are, perhaps they have a different hardware queue or so? That'd be the case for Intel - different hardware queue, same (or at least similar) protocol spoken for the DMA hardware itself, but different contents of the messages obviously.

I want to be careful talking about "control" but for IPA it comes from user space. For the purpose of getting initial code upstream, all of that control functionality (which was IOCTL based) has been removed, and a fixed configuration is assumed.

But something that's ioctl based is just one form of "control" pathway. I was thinking of the AT or MBIM commands "control" channel. And then ioctls are likely something that terminates in the *driver*, right? I mean, the driver wouldn't get an ioctl and actually talk AT commands to the device ...

But yes, the various control planes are confusing, we need to disentangle that. I tried over in the other email by layering where the control terminates.

johannes

reviewer
#70 lore raw
In reply to Arnd Bergmann
Quoted 10 lines · Arnd Bergmann
Quoted 6 lines

The IPA hardware is actually something that sits *between* the AP and the modem. It implements one form of communication pathway (IP data), but there are others (including QMI, which presents a network-like interface but it's actually implemented via clever use of shared memory and interrupts).

Can you clarify how QMI fits in here? Do you mean one has to talk to both IPA and QMI to use the modem, or are these two alternative implementations for the same basic purpose?

I'm not going to comment on QMI specifically, because my understanding might well be wrong, and any response to your question will likely correct my understanding :-)

(Thus, you should probably also ignore everything I ever said about QMI)

Quoted 6 lines

My previous understanding was that from the hardware perspective there is only one control interface, which is for IPA. Part of this is abstracted to user space with ioctl commands to the IPA driver, and then one must set up rmnet to match these by configuring an rmnet device over netlink messages from user space, but rmnet does not have a control protocol with the hardware.

Right so this is why I say it's confusing when we just talk about "control interface" or "path".

I see multiple layers of control

* hardware control, which you mention here. This might be things like "enable/disable aggregation on an rmnet channel" etc. I guess this type of thing would have been implemented with ioctls? Not the aggregation specifically, but things that affect how you set up the hardware.

* modem control, which we conflate, but can be like AT commands or MBIM. From the kernel driver POV, this is actually just another channel it provides for userspace to talk to the modem.

johannes

reviewer
#54 lore raw
In reply to Alex Elder
Quoted 5 lines · Alex Elder

We need to identify the existence of a WWAN device (which is I guess--typically? always?--a modem). Perhaps that can be discovered in some cases but I think it means a node described by Device Tree.

Yeah, perhaps that's something you could do. I'm not sure though. For one, for USB devices, obviously it isn't :-) And even for IPA you might want to support existing DTs I guess.

Quoted 5 lines

So you're saying you have a single Ethernet driver, and it can drive an Ethernet device connected to a WWAN, or not connected to a WWAN, without any changes. The only distinction is that if the device is part of a WWAN it needs to register with the WWAN framework. Is that right?

That's what I'm thinking, and I believe (mostly from discussions with Dan) that this actually exists.

Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

So maybe:
- Hardware probe detects a WWAN device
- The drivers that detect the WWAN device register it with the WWAN core code.
- A control channel is instantiated at/before the time the WWAN device is registered
- Something in user space should manage the bring-up of any other things on the WWAN device thereafter

But those things need to actually get connected first :-)

Quoted 7 lines

What I meant is that the registering with the "WWAN core code" is what does that "connecting." The WWAN code has the information about what got registered. But as I said above, this WWAN device needs to be identified, and I think (at least for IPA) that will require something in Device Tree. That will "connect" them.

Or I might be misunderstanding your point.

No, I think we're mostly agreeing, just thinking about different scenarios. I think for IPA you don't really *need* anything in the DT though - as soon as the IPA driver is loaded you know for sure you actually have a modem there, and the IPA driver presumably loads based on some existing probing (didn't look at it now).

Now, I don't know how the QMI channel to the modem is set up, so of course you'd want a way of identifying that the two channels (IPA and QMI) go to the same device and link them together in the WWAN framework.

Quoted 7 lines
Quoted 4 lines

If userspace actually had the ability to create (data) channels, then it would have the ability to also remove them. Right now, this may or may not be supported by the drivers that act together to form the interfaces to a WWAN device.

I think this (user space control) needs to be an option, but it doesn't have to be the only way.

Agree.

johannes

reviewer
#103 lore raw

Re: WWAN Controller Framework (was IPA [PATCH v2 00/17])

In reply to Alex Elder
Quoted 8 lines · Alex Elder
Quoted 2 lines

I think here we need to be more careful. I don't know how you want to call it, but we actually have multiple levels of control here.

I completely agree with you. From what I understand there exists a control channel (or even more than one?) that serves a very specific purpose in modem management. The main reason I mention the WWAN control function is that someone (maybe you) indicated that a control channel automatically gets created.

It may or may not, right. I just bought a cheap used USB modem, and it just comes with two USB TTY channels, so I guess for data it does PPP on top of that. But those channels are created automatically once you connect the device to the system.

OTOH, for something like the Intel modem, we might well decide not to create *any* channels on driver load, since you have the option of using AT commands or MBIM (but I believe both at the same time wouldn't really make sense, if even allowed).

Quoted 9 lines
Quoted 3 lines

This ... depends a bit on how you exactly define a physical channel here. Is that, to you, the PCIe/USB link? In that case, yes, obviously you have only one physical channel for each WWAN unit.

I think that was what I was trying to capture. There exists one or more "physical" communication paths between the AP and WWAN unit/modem. And while one path *could* carry just one type of traffic, it could also carry multiple logical channels of traffic by multiplexing.

Right.

(What I wasn't aware is that QMI is actually a different physical path. I thought it was just a protocol multiplexed on top of the same IPA physical path).

Quoted 4 lines

I don't think I have any argument with this. I'm going to try to put together something that goes beyond what I wrote in this message, to try to capture what I think we agree on in a sort of loose design document.

Awesome, thanks a lot!

johannes

author
#5 lore raw

Re: [PATCH v2 02/17] dt-bindings: soc: qcom: add IPA bindings

In reply to Alex Elder
Quoted 22 lines · Alex Elder
Quoted 10 lines · Rob Herring
Quoted 9 lines · Alex Elder

Add the binding definitions for the "qcom,ipa" device tree node.

---
 .../devicetree/bindings/net/qcom,ipa.yaml     | 180 ++++++++++++++++++
 1 file changed, 180 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml
Quoted 10 lines

Documentation/devicetree/bindings/net/qcom,ipa.yaml line 1

Quoted 7 lines
Documentation/devicetree/bindings/net/qcom,ipa.yaml new file +1 -0
new file mode 100644
index 000000000000..0037fc278a61
@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: GPL-2.0

New bindings are preferred to be dual GPL-2.0 and BSD-2-Clause. But that's really a decision for the submitter.

Quoted 4 lines

Thanks Rob. I'll ask Qualcomm if there's any problem with doing that; I presume not. If I re-submit this with dual copyright, I will include your Reviewed-by despite the change, OK?

FYI I have the go-ahead to use dual GPL-2.0 and BSD-2-Clause bindings on this, and will mark it that way whenever I next post this code for review.

I will also be updating other Qualcomm bindings to have a dual copyright (in a separate series). We'll want to get an ack from appropriate Code Aurora developers on those (I'll provide more detail at the time those get posted).

-Alex

Quoted 6 lines

-Alex

Quoted 3 lines