Skip to contentSkip to conversation

misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

PATCH Superseded Ravi Patel 4 participants 29 messages Dec 21, 2013 – Jan 30, 2014

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ravi Patel

author
#0 lore raw

v2 cover misc: xgene: Add support for APM X-Gene SoC Queue Manager/Traffic Manager

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

V2:
* Adding COMPILE_TEST dependency for QMTM in Kconfig
* Updated license banner as per review comments
* Fixed alignment for macros
* Changed EXPORT_SYMBOL to EXPORT_SYMBOL_GPL

V1:
* inital version

---
Ravi Patel (4):
Documentation: Add documentation for APM X-Gene SoC Queue
Manager/Traffic Manager DTS binding
misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic
Manager
arm64: boot: dts: Add DTS entries for APM X-Gene SoC Queue
Manager/Traffic Manager
misc: xgene: Add error handling for APM X-Gene SoC Queue
Manager/Traffic Manager

 .../devicetree/bindings/misc/apm-xgene-qmtm.txt    |   51 ++
 MAINTAINERS                                        |    7 +
 arch/arm64/boot/dts/apm-storm.dtsi                 |   17 +
 drivers/misc/Kconfig                               |    1 +
 drivers/misc/Makefile                              |    1 +
 drivers/misc/xgene/Kconfig                         |    1 +
 drivers/misc/xgene/Makefile                        |    5 +
 drivers/misc/xgene/qmtm/Kconfig                    |    9 +
 drivers/misc/xgene/qmtm/Makefile                   |    7 +
 drivers/misc/xgene/qmtm/xgene_qmtm_error.c         |  283 ++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.c          |  765 ++++++++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.h          |  138 ++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c         |  370 ++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h         |  139 ++++
 include/misc/xgene/xgene_qmtm.h                    |  277 +++++++
 15 files changed, 2071 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
 create mode 100644 drivers/misc/xgene/Kconfig
 create mode 100644 drivers/misc/xgene/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/Kconfig
 create mode 100644 drivers/misc/xgene/qmtm/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_error.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.h
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h
 create mode 100644 include/misc/xgene/xgene_qmtm.h
Signature · 1 line
1.7.9.5
author
#1 lore raw

PATCH v2 1/4 Documentation: Add documentation for APM X-Gene SoC Queue Manager/Traffic Manager DTS binding

This patch adds device tree binding documentation for APM X-Gene SoC
Queue Manager/Traffic Manager.

---
 .../devicetree/bindings/misc/apm-xgene-qmtm.txt    |   51 ++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt new file +51 -0
new file mode 100644
index 0000000..c3fcbd2
@@ -0,0 +1,51 @@
+* APM X-Gene SoC Queue Manager/Traffic Manager nodes
+
+QMTM nodes are defined to describe on-chip Queue Managers in APM X-Gene SoC.
+APM X-Gene SoC Ethernet, PktDMA (XOR Engine), and Security Engine subsystems
+communicate with a central Queue Manager using messages which include
+information about the work to be performed and the location of the associated
+data buffers. There are multiple instances of QMTM. Each QMTM instance has its
+own node. Its corresponding clock nodes are shown below.
+
+Required properties:
+- compatible		: Shall be "apm,xgene-qmtm-lite" for QMLite instance
+- reg			: First memory resource shall be the QMTM CSR memory
+			  resource.
+			  Second memory resource shall be the QMTM IO-Fabric
+			  memory resource.
+- slave-name		: Shall be "CPU_QMTM3" which is receiver for ingress
+			  work messages for the QMTM. Here receiver is CPU.
+- interrupts		: First interrupt resource shall be the QMTM Error
+			  interrupt.
+			  Remaining interrupt resources shall be the Ingress
+			  work message interrupt mapping for receiver,
+			  receiving work messages for the QMTM.
+- clocks		: Reference to the clock entry.
+
+Optional properties:
+- status		: Shall be "ok" if enabled or "disabled" if disabled.
+			  Default is "ok".
+
+Example:
+		qmlclk: qmlclk {
+			compatible = "apm,xgene-device-clock";
+			#clock-cells = <1>;
+			clock-names = "qmlclk";
+			status = "ok";
+			csr-offset = <0x0>;
+			csr-mask = <0x3>;
+			enable-offset = <0x8>;
+			enable-mask = <0x3>;
+		};
+
+		qmlite: qmtm at 17030000 {
+			compatible = "apm,xgene-qmtm-lite";
+			reg = <0x0 0x17030000 0x0 0x10000>,
+			      <0x0 0x10000000 0x0 0x400000>;
+			slave-name = "CPU_QMTM3";
+			interrupts = <0x0 0x40 0x4>,
+				     <0x0 0x3c 0x4>;
+			status = "ok";
+			#clock-cells = <1>;
+			clocks = <&qmlclk 0>;
+		};
Signature · 1 line
1.7.9.5
maintainer
#2 lore raw

Re: [PATCH V2 1/4] Documentation: Add documentation for APM X-Gene SoC Queue Manager/Traffic Manager DTS binding

Quoted 15 lines · Ravi Patel
 create mode 100644 Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt
Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt new file +7 -0
new file mode 100644
index 0000000..c3fcbd2
@@ -0,0 +1,51 @@
+* APM X-Gene SoC Queue Manager/Traffic Manager nodes
+
+QMTM nodes are defined to describe on-chip Queue Managers in APM X-Gene SoC.
+APM X-Gene SoC Ethernet, PktDMA (XOR Engine), and Security Engine subsystems
+communicate with a central Queue Manager using messages which include
+information about the work to be performed and the location of the associated
+data buffers. There are multiple instances of QMTM. Each QMTM instance has its

Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt line 17

Quoted 10 lines
Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt new file +10 -0
+own node. Its corresponding clock nodes are shown below.
+
+Required properties:
+- compatible		: Shall be "apm,xgene-qmtm-lite" for QMLite instance
+- reg			: First memory resource shall be the QMTM CSR memory
+			  resource.
+			  Second memory resource shall be the QMTM IO-Fabric
+			  memory resource.
+- slave-name		: Shall be "CPU_QMTM3" which is receiver for ingress
+			  work messages for the QMTM. Here receiver is CPU.

What is that string used for? Who decides what strings are valid in the future? Why does the device even need to know the name of the receiver, shouldn't that have a separate node?

Quoted 12 lines
Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt new file +12 -0
+- interrupts		: First interrupt resource shall be the QMTM Error
+			  interrupt.
+			  Remaining interrupt resources shall be the Ingress
+			  work message interrupt mapping for receiver,
+			  receiving work messages for the QMTM.
+- clocks		: Reference to the clock entry.
+
+Optional properties:
+- status		: Shall be "ok" if enabled or "disabled" if disabled.
+			  Default is "ok".
+
+Example:

Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt line 39

Quoted 10 lines
Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt new file +10 -0
+		qmlclk: qmlclk {
+			compatible = "apm,xgene-device-clock";
+			#clock-cells = <1>;
+			clock-names = "qmlclk";
+			status = "ok";
+			csr-offset = <0x0>;
+			csr-mask = <0x3>;
+			enable-offset = <0x8>;
+			enable-mask = <0x3>;
+		};

This looks like an invalid node, which makes a bad example. Why do you have a "clock-names" entry here but no "clocks"?

Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt line 51

Quoted 11 lines
Documentation/devicetree/bindings/misc/apm-xgene-qmtm.txt new file +11 -0
+		qmlite: qmtm at 17030000 {
+			compatible = "apm,xgene-qmtm-lite";
+			reg = <0x0 0x17030000 0x0 0x10000>,
+			      <0x0 0x10000000 0x0 0x400000>;
+			slave-name = "CPU_QMTM3";
+			interrupts = <0x0 0x40 0x4>,
+				     <0x0 0x3c 0x4>;
+			status = "ok";
+			#clock-cells = <1>;
+			clocks = <&qmlclk 0>;
+		};

#clock-cells is neither a required nor an optional property for "apm,xgene-qmtm-lite" according to the documentation above. Please make the example match the specification.

Arnd

author
#3 lore raw

PATCH v2 2/4 misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

In reply to Ravi Patel

This patch adds APM X-Gene SoC Queue Manager/Traffic Manager base driver.
QMTM is requried by Ethernet, PktDMA (XOR Engine) and Security Engine subsystems.

---
 MAINTAINERS                                |    7 +
 drivers/misc/Kconfig                       |    1 +
 drivers/misc/Makefile                      |    1 +
 drivers/misc/xgene/Kconfig                 |    1 +
 drivers/misc/xgene/Makefile                |    5 +
 drivers/misc/xgene/qmtm/Kconfig            |    9 +
 drivers/misc/xgene/qmtm/Makefile           |    7 +
 drivers/misc/xgene/qmtm/xgene_qmtm_main.c  |  761 ++++++++++++++++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.h  |  134 +++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c |  370 ++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h |  139 +++++
 include/misc/xgene/xgene_qmtm.h            |  277 ++++++++++
 12 files changed, 1712 insertions(+)
 create mode 100644 drivers/misc/xgene/Kconfig
 create mode 100644 drivers/misc/xgene/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/Kconfig
 create mode 100644 drivers/misc/xgene/qmtm/Makefile
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_main.h
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.c
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_storm.h
 create mode 100644 include/misc/xgene/xgene_qmtm.h
MAINTAINERS +7 -0
index f216db8..920cae8 100644
@@ -665,6 +665,13 @@ S:	Maintained
 F:	drivers/net/appletalk/
 F:	net/appletalk/
 
+APPLIEDMICRO (APM) X-GENE SOC QUEUE MANAGER/TRAFFIC MANAGER (QMTM) DRIVER
+M:	Ravi Patel <[email protected]>
+M:	Keyur Chudgar <[email protected]>
+S:	Maintained
+F:	drivers/misc/xgene/
+F:	include/misc/xgene/xgene_qmtm.h
+
 APTINA CAMERA SENSOR PLL
 M:	Laurent Pinchart <[email protected]>
 L:	linux-media at vger.kernel.org
drivers/misc/Kconfig +1 -0
index a3e291d..b309553 100644
@@ -525,4 +525,5 @@ source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
 source "drivers/misc/mic/Kconfig"
+source "drivers/misc/xgene/Kconfig"
 endmenu
drivers/misc/Makefile +1 -0
index f45473e..0fd3b1b 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
 obj-y				+= mic/
+obj-$(CONFIG_ARCH_XGENE)	+= xgene/
drivers/misc/xgene/Kconfig new file +1 -0
new file mode 100644
index 0000000..8f38568
@@ -0,0 +1 @@
+source "drivers/misc/xgene/qmtm/Kconfig"
drivers/misc/xgene/Makefile new file +5 -0
new file mode 100644
index 0000000..198c2e6
@@ -0,0 +1,5 @@
+#
+# Makefile for APM X-GENE misc drivers.
+#
+
+obj-$(CONFIG_XGENE_QMTM)	+= qmtm/
drivers/misc/xgene/qmtm/Kconfig new file +9 -0
new file mode 100644
index 0000000..8efda4d
@@ -0,0 +1,9 @@
+config XGENE_QMTM
+	tristate "APM X-Gene Queue Manager/Traffic Manager driver"
+	depends on ARM64 || COMPILE_TEST
+	default y
+	help
+	  This option enables APM X-Gene Queue Manager Traffic Manager (QMTM)
+	  driver support.
+	  QMTM is required for Ethernet, PktDMA (XOR Engine) and Security
+	  Engine.
drivers/misc/xgene/qmtm/Makefile new file +7 -0
new file mode 100644
index 0000000..68c2a86
@@ -0,0 +1,7 @@
+#
+# Makefile for APM X-GENE Queue Manager Traffic Manager driver.
+#
+
+obj-$(CONFIG_XGENE_QMTM) += xgene-qmtm.o
+
+xgene-qmtm-objs := xgene_qmtm_main.o xgene_qmtm_storm.o
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +761 -0

761 lines not on this page · raw message

drivers/misc/xgene/qmtm/xgene_qmtm_main.h new file +134 -0

134 lines not on this page · raw message

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +370 -0

370 lines not on this page · raw message

drivers/misc/xgene/qmtm/xgene_qmtm_storm.h new file +139 -0

139 lines not on this page · raw message

include/misc/xgene/xgene_qmtm.h new file +277 -0

277 lines not on this page · raw message

Signature · 1 line
1.7.9.5
maintainer
#4 lore raw

Re: [PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

Quoted 10 lines · Ravi Patel
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +10 -0
+/* CSR Address Macros */
+#define CSR_QM_CONFIG_ADDR	0x00000004
+#define  QM_ENABLE_WR(src)	(((u32)(src)<<31) & 0x80000000)
+
+#define CSR_PBM_ADDR		0x00000008
+#define  OVERWRITE_WR(src)	(((u32)(src)<<31) & 0x80000000)
+#define  SLVID_PBN_WR(src)	(((u32)(src)) & 0x000003ff)
+#define  SLAVE_ID_SHIFT		6
+
+#define CSR_PBM_BUF_WR_ADDR	0x0000000c

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 48

Quoted 10 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +10 -0
+#define CSR_PBM_BUF_RD_ADDR	0x00000010
+#define  PB_SIZE_WR(src)	(((u32)(src)<<31) & 0x80000000)
+#define  PREFETCH_BUF_EN_SET(dst, src) \
+	(((dst) & ~0x00200000) | (((u32)(src)<<21) & 0x00200000))
+#define  IS_FREE_POOL_SET(dst, src) \
+	(((dst) & ~0x00100000) | (((u32)(src)<<20) & 0x00100000))
+#define  TLVQ_SET(dst, src) \
+	(((dst) & ~0x00080000) | (((u32)(src)<<19) & 0x00080000))
+#define  CORRESPONDING_QNUM_SET(dst, src) \
+	(((dst) & ~0x0007fe00) | (((u32)(src)<<9) & 0x0007fe00))

These macros are rather unreadable and unhelpful. Can't you just open-code them in the users?

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 58

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+#define CSR_THRESHOLD0_SET1_ADDR	0x00000030
+#define CSR_THRESHOLD1_SET1_ADDR	0x00000034
+#define CSR_HYSTERESIS_ADDR		0x00000068
+#define CSR_QM_MBOX_NE_INT_MODE_ADDR	0x0000017c
+#define CSR_QMLITE_PBN_MAP_0_ADDR	0x00000228

What about all the other registers inbetween?

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 81

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data)
+{
+	void *addr = (u8 *)qmtm->csr_vaddr + offset;
+	writel(data, addr);
+}

Type mismatch: writel requires a "void __iomem*" pointer. Please check your driver using "sparse" to find bugs like this.

Quoted 5 lines

+/**
+ * xgene_qmtm_set_qinfo - Create and configure a queue
+ * @sdev: Slave context
+ * @qtype: Queue type (P_QUEUE or FP_QUEUE)
+ * @qsize: Queue size see xgene_qmtm_qsize

Quoted 8 lines

+ * @qaccess: Queue access method see xgene_qmtm_qaccess
+ * @flags: Queue Information flags
+ * @qpaddr: If Desire Queue Physical Address to use
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to create and configure a queue.
+ *
+ * Return: 0 on Success or -1 on Failure

-1 on failure is not an appropriate return code. Please use standard error numbers such as -EINVAL.

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 692

Quoted 11 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +11 -0
+	rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
+	if (rc < 0) {
+		dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
+		return rc;
+	}
+
+	sdev = xgene_qmtm_get_sdev((char *)name);
+	if (sdev == NULL) {
+		dev_err(&pdev->dev, "Ingress Slave error\n");
+		return -ENODEV;
+	}

Why are you referring to another device by a string? Shouldn't that be a phandle?

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 744

Quoted 9 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +9 -0
+static struct of_device_id xgene_qmtm_match[] = {
+	{.compatible = "apm,xgene-qmtm-xge0",},
+	{.compatible = "apm,xgene-qmtm-soc",},
+	{.compatible = "apm,xgene-qmtm-xge2",},
+	{.compatible = "apm,xgene-qmtm-lite",},
+	{},
+};
+
+MODULE_DEVICE_TABLE(of, xgene_qmtm_match);

You don't seem to differentiate between the four different strings. Why can't they just all be compatible to "apm,xgene-qmtm"?

drivers/misc/xgene/qmtm/xgene_qmtm_main.h line 126

Quoted 3 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.h new file +3 -0
+
+#define VIRT_TO_PHYS(x)	virt_to_phys(x)
+#define PHYS_TO_VIRT(x)	phys_to_virt(x)

These don't belong here, use the standard functions. Also, it seems unlikely that a device driver actually wants physical addresses. Are these used for DMA?

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 43

Quoted 12 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +12 -0
+static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
+	[SLAVE_ETH0] = {
+			.name = "SGMII0",
+			.compatible = "apm,xgene-qmtm-soc",
+			.slave = SLAVE_ETH0,
+			.qmtm_ip = QMTM1,
+			.slave_id = QMTM_SLAVE_ID_ETH0,
+			.wq_pbn_start = 0x00,
+			.wq_pbn_count = 0x08,
+			.fq_pbn_start = 0x20,
+			.fq_pbn_count = 0x08,
+			},

This table is backwards: the qmtm driver provides services to client drivers, it should have zero knowledge about what the clients are.

Remove this array here and put the data into properties of the client drivers.

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 302

Quoted 13 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +13 -0
+	if (qinfo->flags & XGENE_SLAVE_Q_ADDR_ALLOC) {
+		qinfo->qdesc->qvaddr = kzalloc(qsize, GFP_KERNEL);
+		if (qinfo->qdesc->qvaddr == NULL) {
+			kfree(qinfo->qdesc);
+			rc = -ENOMEM;
+			goto _ret_set_qstate;
+		}
+
+		qinfo->qpaddr = (u64) VIRT_TO_PHYS(qinfo->qdesc->qvaddr);
+	} else {
+		qinfo->qdesc->qvaddr = PHYS_TO_VIRT(qinfo->qpaddr);
+		memset(qinfo->qdesc->qvaddr, 0, qsize);
+	}

This looks a lot like you are doing DMA. Please use the appropriate DMA abstractions for this.

Quoted 12 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +12 -0
+struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
+{
+	struct xgene_qmtm *qmtm = NULL;
+	struct xgene_qmtm_sdev *sdev = NULL;
+	struct device_node *np = NULL;
+	struct platform_device *platdev;
+	u8 slave;
+
+	for (slave = 0; slave < SLAVE_MAX; slave++) {
+		sdev = &storm_sdev[slave];
+		if (sdev->name && strcmp(name, sdev->name) == 0) {
+			np = of_find_compatible_node(NULL, NULL,

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 357

Quoted 10 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +10 -0
+				sdev->compatible);
+			break;
+		}
+	}
+
+	if (np == NULL)
+		return NULL;
+
+	platdev = of_find_device_by_node(np);
+	qmtm = platform_get_drvdata(platdev);

Get rid of the of_find_compatible_node() here, this is another indication that you are doing things backwards.

Quoted 14 lines
include/misc/xgene/xgene_qmtm.h new file +14 -0
+/* QMTM messaging structures */
+/* 16 byte QMTM message format */
+struct xgene_qmtm_msg16 {
+	u32 UserInfo;
+
+	u32 FPQNum:12;
+	u32 Rv2:2;
+	u32 ELErr:2;
+	u32 LEI:2;
+	u32 NV:1;
+	u32 LL:1;
+	u32 PB:1;
+	u32 HB:1;
+	u32 Rv:1;

include/misc/xgene/xgene_qmtm.h line 98

Quoted 10 lines
include/misc/xgene/xgene_qmtm.h new file +10 -0
+	u32 IN:1;
+	u32 RType:4;
+	u32 LErr:3;
+	u32 HL:1;
+
+	u64 DataAddr:42;	/* 32/10 split */
+	u32 Rv6:6;
+	u32 BufDataLen:15;
+	u32 C:1;
+} __packed;

Bitfields are generally not endian-safe. Better use raw u32 values and bit masks. Also don't use __packed in hardware descriptors, as it messes up atomicity of accesses.

Arnd

author
#5 lore raw

Re: [PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

Quoted 16 lines · Arnd Bergmann
Quoted 15 lines · Ravi Patel
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +15 -0
+/* CSR Address Macros */
+#define CSR_QM_CONFIG_ADDR   0x00000004
+#define  QM_ENABLE_WR(src)   (((u32)(src)<<31) & 0x80000000)
+
+#define CSR_PBM_ADDR         0x00000008
+#define  OVERWRITE_WR(src)   (((u32)(src)<<31) & 0x80000000)
+#define  SLVID_PBN_WR(src)   (((u32)(src)) & 0x000003ff)
+#define  SLAVE_ID_SHIFT              6
+
+#define CSR_PBM_BUF_WR_ADDR  0x0000000c
+#define CSR_PBM_BUF_RD_ADDR  0x00000010
+#define  PB_SIZE_WR(src)     (((u32)(src)<<31) & 0x80000000)
+#define  PREFETCH_BUF_EN_SET(dst, src) \
+     (((dst) & ~0x00200000) | (((u32)(src)<<21) & 0x00200000))
+#define  IS_FREE_POOL_SET(dst, src) \
Quoted 8 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 48

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+     (((dst) & ~0x00100000) | (((u32)(src)<<20) & 0x00100000))
+#define  TLVQ_SET(dst, src) \
+     (((dst) & ~0x00080000) | (((u32)(src)<<19) & 0x00080000))
+#define  CORRESPONDING_QNUM_SET(dst, src) \
+     (((dst) & ~0x0007fe00) | (((u32)(src)<<9) & 0x0007fe00))

These macros are rather unreadable and unhelpful. Can't you just open-code them in the users?

Sure, will take of this in next patch.

Quoted 7 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 58

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+#define CSR_THRESHOLD0_SET1_ADDR     0x00000030
+#define CSR_THRESHOLD1_SET1_ADDR     0x00000034
+#define CSR_HYSTERESIS_ADDR          0x00000068
+#define CSR_QM_MBOX_NE_INT_MODE_ADDR 0x0000017c
+#define CSR_QMLITE_PBN_MAP_0_ADDR    0x00000228

What about all the other registers in between?

All the other registers in between needs no change in values, default values are fine.
These registers are not accessed and so we didn't defined them. Let us know if you need them for a reason.

Quoted 8 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 81

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data)
+{
+     void *addr = (u8 *)qmtm->csr_vaddr + offset;
+     writel(data, addr);
+}

Type mismatch: writel requires a "void __iomem*" pointer. Please check your driver using "sparse" to find bugs like this.

Sure, will take care of this in next patch

Quoted 14 lines
Quoted 5 lines

+/**
+ * xgene_qmtm_set_qinfo - Create and configure a queue

+ * @sdev:    Slave context
+ * @qtype:   Queue type (P_QUEUE or FP_QUEUE)
+ * @qsize:   Queue size see xgene_qmtm_qsize
Quoted 8 lines

+ * @qaccess: Queue access method see xgene_qmtm_qaccess
+ * @flags: Queue Information flags
+ * @qpaddr: If Desire Queue Physical Address to use
+ *
+ * This API will be called by APM X-Gene SOC Ethernet, PktDMA (XOR Engine),
+ * and Security Engine subsystems to create and configure a queue.
+ *
+ * Return: 0 on Success or -1 on Failure

Quoted 2 lines

-1 on failure is not an appropriate return code. Please use standard error numbers such as -EINVAL.

Sure, will take care of this in next patch

Quoted 12 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 692

Quoted 11 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +11 -0
+     rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
+     if (rc < 0) {
+             dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
+             return rc;
+     }
+
+     sdev = xgene_qmtm_get_sdev((char *)name);
+     if (sdev == NULL) {
+             dev_err(&pdev->dev, "Ingress Slave error\n");
+             return -ENODEV;
+     }
Quoted 1 line

Why are you referring to another device by a string? Shouldn't that be a phandle?

Let me go through this and find the possibility.

Quoted 4 lines
Quoted 4 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +4 -0
+static struct of_device_id xgene_qmtm_match[] = {
+     {.compatible = "apm,xgene-qmtm-xge0",},
+     {.compatible = "apm,xgene-qmtm-soc",},
+     {.compatible = "apm,xgene-qmtm-xge2",},
Quoted 8 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 744

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+     {.compatible = "apm,xgene-qmtm-lite",},
+     {},
+};
+
+MODULE_DEVICE_TABLE(of, xgene_qmtm_match);

You don't seem to differentiate between the four different strings. Why can't they just all be compatible to "apm,xgene-qmtm"?

All the QMTM manages resources for different subsystems.
So all the 4 QMTM are different, so each of them should have a unique identity.

Quoted 6 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.h line 126

Quoted 3 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.h new file +3 -0
+
+#define VIRT_TO_PHYS(x)      virt_to_phys(x)
+#define PHYS_TO_VIRT(x)      phys_to_virt(x)

These don't belong here, use the standard functions. Also, it seems unlikely that a device driver actually wants physical addresses. Are these used for DMA?

The memory addresses passed in this macro/function are not used for DMA. For creating a physical queue in the QMTM hardware, it needs the physical address
of the queue kmalloc'ed by the driver,

Quoted 13 lines

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 43

Quoted 12 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +12 -0
+static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
+     [SLAVE_ETH0] = {
+                     .name = "SGMII0",
+                     .compatible = "apm,xgene-qmtm-soc",
+                     .slave = SLAVE_ETH0,
+                     .qmtm_ip = QMTM1,
+                     .slave_id = QMTM_SLAVE_ID_ETH0,
+                     .wq_pbn_start = 0x00,
+                     .wq_pbn_count = 0x08,
+                     .fq_pbn_start = 0x20,
+                     .fq_pbn_count = 0x08,
+                     },
Quoted 5 lines

This table is backwards: the qmtm driver provides services to client drivers, it should have zero knowledge about what the clients are.

Remove this array here and put the data into properties of the client drivers.

QMTM is managing the queue resources in its hardware for all the clients. The clients meaning, Ethernet, PktDMA (XOR Engine) and Security Engine, has zero knowledge of its qm resources.
So for that reason this array defining QMTM's client properties is defined in QMTM driver

Quoted 14 lines

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 302

Quoted 13 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +13 -0
+     if (qinfo->flags & XGENE_SLAVE_Q_ADDR_ALLOC) {
+             qinfo->qdesc->qvaddr = kzalloc(qsize, GFP_KERNEL);
+             if (qinfo->qdesc->qvaddr == NULL) {
+                     kfree(qinfo->qdesc);
+                     rc = -ENOMEM;
+                     goto _ret_set_qstate;
+             }
+
+             qinfo->qpaddr = (u64) VIRT_TO_PHYS(qinfo->qdesc->qvaddr);
+     } else {
+             qinfo->qdesc->qvaddr = PHYS_TO_VIRT(qinfo->qpaddr);
+             memset(qinfo->qdesc->qvaddr, 0, qsize);
+     }
Quoted 2 lines

This looks a lot like you are doing DMA. Please use the appropriate DMA abstractions for this.

Sure, will take are for the DMA abstraction in next revision.

Quoted 17 lines
Quoted 17 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +17 -0
+struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
+{
+     struct xgene_qmtm *qmtm = NULL;
+     struct xgene_qmtm_sdev *sdev = NULL;
+     struct device_node *np = NULL;
+     struct platform_device *platdev;
+     u8 slave;
+
+     for (slave = 0; slave < SLAVE_MAX; slave++) {
+             sdev = &storm_sdev[slave];
+             if (sdev->name && strcmp(name, sdev->name) == 0) {
+                     np = of_find_compatible_node(NULL, NULL,
+                             sdev->compatible);
+                     break;
+             }
+     }
+
Quoted 8 lines

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 357

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +5 -0
+     if (np == NULL)
+             return NULL;
+
+     platdev = of_find_device_by_node(np);
+     qmtm = platform_get_drvdata(platdev);

Get rid of the of_find_compatible_node() here, this is another indication that you are doing things backwards.

Since QMTM is managing resource for its client, we have defined the mapping in the QMTM driver.
So whenever QMTM client like Ethernet driver probe is called, it will pass on its ID/slave-name
to get its context which is managed by QMTM

Quoted 20 lines
Quoted 20 lines
include/misc/xgene/xgene_qmtm.h new file +20 -0
+/* QMTM messaging structures */
+/* 16 byte QMTM message format */
+struct xgene_qmtm_msg16 {
+     u32 UserInfo;
+
+     u32 FPQNum:12;
+     u32 Rv2:2;
+     u32 ELErr:2;
+     u32 LEI:2;
+     u32 NV:1;
+     u32 LL:1;
+     u32 PB:1;
+     u32 HB:1;
+     u32 Rv:1;
+     u32 IN:1;
+     u32 RType:4;
+     u32 LErr:3;
+     u32 HL:1;
+
+     u64 DataAddr:42;        /* 32/10 split */
Quoted 8 lines

include/misc/xgene/xgene_qmtm.h line 98

Quoted 4 lines
include/misc/xgene/xgene_qmtm.h new file +4 -0
+     u32 Rv6:6;
+     u32 BufDataLen:15;
+     u32 C:1;
+} __packed;

Bitfields are generally not endian-safe. Better use raw u32 values and bit masks. Also don't use __packed in hardware descriptors, as it messes up atomicity of accesses.

I explained in another email the reason for using bit fields. Let me know if you want me to paste it here again.

Thanks,
Ravi

maintainer
#6 lore raw

Re: [PATCH V2 2/4] misc: xgene: Add base driver for APM X-Gene SoC Queue Manager/Traffic Manager

Quoted 11 lines · On Saturday 21 December 2013 17:45:30 Ravi Patel wrote:
Quoted 9 lines · Arnd Bergmann

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 58

Quoted 5 lines · Ravi Patel
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+#define CSR_THRESHOLD0_SET1_ADDR     0x00000030
+#define CSR_THRESHOLD1_SET1_ADDR     0x00000034
+#define CSR_HYSTERESIS_ADDR          0x00000068
+#define CSR_QM_MBOX_NE_INT_MODE_ADDR 0x0000017c
+#define CSR_QMLITE_PBN_MAP_0_ADDR    0x00000228

What about all the other registers in between?

Quoted 4 lines

All the other registers in between needs no change in values, default values are fine.
These registers are not accessed and so we didn't defined them. Let us know if you need them for a reason.

Ok, this is fine. I was just making sure there isn't another driver that talks to these registers.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +11 -0
+     rc = of_property_read_string(pdev->dev.of_node, "slave-name", &name);
+     if (rc < 0) {
+             dev_err(&pdev->dev, "Failed to get QMTM Ingress slave-name\n");
+             return rc;
+     }
+
+     sdev = xgene_qmtm_get_sdev((char *)name);
+     if (sdev == NULL) {
+             dev_err(&pdev->dev, "Ingress Slave error\n");
+             return -ENODEV;
+     }
Quoted 3 lines
Quoted 1 line

Why are you referring to another device by a string? Shouldn't that be a phandle?

Let me go through this and find the possibility.

To clarify, I think the slave driver should look up a phandle to find the qmtm device.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +4 -0
+static struct of_device_id xgene_qmtm_match[] = {
+     {.compatible = "apm,xgene-qmtm-xge0",},
+     {.compatible = "apm,xgene-qmtm-soc",},
+     {.compatible = "apm,xgene-qmtm-xge2",},
Quoted 8 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.c line 744

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.c new file +5 -0
+     {.compatible = "apm,xgene-qmtm-lite",},
+     {},
+};
+
+MODULE_DEVICE_TABLE(of, xgene_qmtm_match);

You don't seem to differentiate between the four different strings. Why can't they just all be compatible to "apm,xgene-qmtm"?

Quoted 2 lines

All the QMTM manages resources for different subsystems.
So all the 4 QMTM are different, so each of them should have a unique identity.

Are you sure it's not just four instances of the same hardware block connected to different slaves?

The "compatible" value should only identify differences in the register set.

Quoted 10 lines
Quoted 5 lines

drivers/misc/xgene/qmtm/xgene_qmtm_main.h line 126

Quoted 2 lines
drivers/misc/xgene/qmtm/xgene_qmtm_main.h new file +2 -0
+#define VIRT_TO_PHYS(x)      virt_to_phys(x)
+#define PHYS_TO_VIRT(x)      phys_to_virt(x)

These don't belong here, use the standard functions. Also, it seems unlikely that a device driver actually wants physical addresses. Are these used for DMA?

The memory addresses passed in this macro/function are not used for DMA. For creating a physical queue in the QMTM hardware, it needs the physical address
of the queue kmalloc'ed by the driver,

Who is accessing the physical address then? Does this get passed to some sort of hypervisor?

Quoted 16 lines
Quoted 16 lines

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 43

Quoted 12 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +12 -0
+static struct xgene_qmtm_sdev storm_sdev[SLAVE_MAX] = {
+     [SLAVE_ETH0] = {
+                     .name = "SGMII0",
+                     .compatible = "apm,xgene-qmtm-soc",
+                     .slave = SLAVE_ETH0,
+                     .qmtm_ip = QMTM1,
+                     .slave_id = QMTM_SLAVE_ID_ETH0,
+                     .wq_pbn_start = 0x00,
+                     .wq_pbn_count = 0x08,
+                     .fq_pbn_start = 0x20,
+                     .fq_pbn_count = 0x08,
+                     },

This table is backwards: the qmtm driver provides services to client drivers, it should have zero knowledge about what the clients are.

Quoted 8 lines
Quoted 2 lines

Remove this array here and put the data into properties of the client drivers.

QMTM is managing the queue resources in its hardware for all the clients. The clients meaning, Ethernet, PktDMA (XOR Engine) and Security Engine, has zero knowledge of its qm resources.
So for that reason this array defining QMTM's client properties is defined in QMTM driver

No, that's the wrong way around. Please have a look at how other subsystems (irq, dmaengine, clock, ...) manage the split between knowledge.

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +17 -0
+struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name)
+{
+     struct xgene_qmtm *qmtm = NULL;
+     struct xgene_qmtm_sdev *sdev = NULL;
+     struct device_node *np = NULL;
+     struct platform_device *platdev;
+     u8 slave;
+
+     for (slave = 0; slave < SLAVE_MAX; slave++) {
+             sdev = &storm_sdev[slave];
+             if (sdev->name && strcmp(name, sdev->name) == 0) {
+                     np = of_find_compatible_node(NULL, NULL,
+                             sdev->compatible);
+                     break;
+             }
+     }
+
Quoted 8 lines

drivers/misc/xgene/qmtm/xgene_qmtm_storm.c line 357

Quoted 5 lines
drivers/misc/xgene/qmtm/xgene_qmtm_storm.c new file +5 -0
+     if (np == NULL)
+             return NULL;
+
+     platdev = of_find_device_by_node(np);
+     qmtm = platform_get_drvdata(platdev);

Get rid of the of_find_compatible_node() here, this is another indication that you are doing things backwards.

Quoted 5 lines

Since QMTM is managing resource for its client, we have defined the mapping in the QMTM driver.
So whenever QMTM client like Ethernet driver probe is called, it will pass on its ID/slave-name
to get its context which is managed by QMTM

The client/slave driver should instead pass the slave ID and other required information, essentially the stuff from the table above, in an abstract form.

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
include/misc/xgene/xgene_qmtm.h new file +20 -0
+/* QMTM messaging structures */
+/* 16 byte QMTM message format */
+struct xgene_qmtm_msg16 {
+     u32 UserInfo;
+
+     u32 FPQNum:12;
+     u32 Rv2:2;
+     u32 ELErr:2;
+     u32 LEI:2;
+     u32 NV:1;
+     u32 LL:1;
+     u32 PB:1;
+     u32 HB:1;
+     u32 Rv:1;
+     u32 IN:1;
+     u32 RType:4;
+     u32 LErr:3;
+     u32 HL:1;
+
+     u64 DataAddr:42;        /* 32/10 split */
Quoted 8 lines

include/misc/xgene/xgene_qmtm.h line 98

Quoted 4 lines
include/misc/xgene/xgene_qmtm.h new file +4 -0
+     u32 Rv6:6;
+     u32 BufDataLen:15;
+     u32 C:1;
+} __packed;

Bitfields are generally not endian-safe. Better use raw u32 values and bit masks. Also don't use __packed in hardware descriptors, as it messes up atomicity of accesses.

Quoted 2 lines

I explained in another email the reason for using bit fields. Let me know if you want me to paste it here again.

No need for that. But driver code really needs to be written in an endian-neutral way if the CPU supports both little-endian and big-endian modes.

Arnd

author
#7 lore raw

PATCH v2 3/4 arm64: boot: dts: Add DTS entries for APM X-Gene SoC Queue Manager/Traffic Manager

In reply to Ravi Patel

This patch adds APM X-Gene SoC Queue Manager/Traffic Manager DTS entries.

---
 arch/arm64/boot/dts/apm-storm.dtsi |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/arm64/boot/dts/apm-storm.dtsi +17 -0
index d37d736..f64f946 100644
@@ -152,6 +152,11 @@
 				reg = <0x0 0x1703C000 0x0 0x1000>;
 				reg-names = "csr-reg";
 				clock-output-names = "qmlclk";
+				status = "ok";
+				csr-offset = <0x0>;
+				csr-mask = <0x3>;
+				enable-offset = <0x8>;
+				enable-mask = <0x3>;
 			};
 
 			ethclk: ethclk {
@@ -187,5 +192,17 @@
 			interrupt-parent = <&gic>;
 			interrupts = <0x0 0x4c 0x4>;
 		};
+
+		qmlite: qmtm at 17030000 {
+			compatible = "apm,xgene-qmtm-lite";
+			reg = <0x0 0x17030000 0x0 0x10000>,
+			      <0x0 0x10000000 0x0 0x400000>;
+			slave-name = "CPU_QMTM3";
+			interrupts = <0x0 0x40 0x4>,
+				     <0x0 0x3c 0x4>;
+			status = "ok";
+			#clock-cells = <1>;
+			clocks = <&qmlclk 0>;
+		};
 	};
 };
Signature · 1 line
1.7.9.5
author
#8 lore raw

PATCH v2 4/4 misc: xgene: Add error handling for APM X-Gene SoC Queue Manager/Traffic Manager

In reply to Ravi Patel

This patch adds support for error handling in APM X-Gene SoC Queue
Manager/Traffic Manager.

---
 drivers/misc/xgene/qmtm/Makefile           |    2 +-
 drivers/misc/xgene/qmtm/xgene_qmtm_error.c |  283 ++++++++++++++++++++++++++++
 drivers/misc/xgene/qmtm/xgene_qmtm_main.c  |    6 +-
 drivers/misc/xgene/qmtm/xgene_qmtm_main.h  |    4 +
 4 files changed, 293 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/xgene/qmtm/xgene_qmtm_error.c
drivers/misc/xgene/qmtm/Makefile +1 -1
index 68c2a86..8448253 100644
@@ -4,4 +4,4 @@
 
 obj-$(CONFIG_XGENE_QMTM) += xgene-qmtm.o
 
-xgene-qmtm-objs := xgene_qmtm_main.o xgene_qmtm_storm.o
+xgene-qmtm-objs := xgene_qmtm_main.o xgene_qmtm_storm.o xgene_qmtm_error.o
drivers/misc/xgene/qmtm/xgene_qmtm_error.c new file +283 -0

283 lines not on this page · raw message

drivers/misc/xgene/qmtm/xgene_qmtm_main.c +5 -1
index cda63e0..833ff82 100644
@@ -588,13 +588,17 @@ static int xgene_qmtm_enable(struct xgene_qmtm *qmtm)
 		qmtm->write_qstate(&qinfo);
 	}
 
-	return rc;
+	/* Enable error reporting */
+	return xgene_qmtm_enable_error(qmtm);
 }
 
 static int xgene_qmtm_disable(struct xgene_qmtm *qmtm)
 {
 	u32 queue_id;
 
+	/* Disable error reporting */
+	xgene_qmtm_disable_error(qmtm);
+
 	for (queue_id = 0; queue_id < qmtm->max_queues; queue_id++) {
 		if (qmtm->qinfo[queue_id]) {
 			dev_err(&qmtm->pdev->dev,
drivers/misc/xgene/qmtm/xgene_qmtm_main.h +4 -0
index d937462..2a60225 100644
@@ -129,6 +129,10 @@ enum xgene_qmtm_fp_mode {
 void xgene_qmtm_wr32(struct xgene_qmtm *qmtm, u32 offset, u32 data);
 void xgene_qmtm_rd32(struct xgene_qmtm *qmtm, u32 offset, u32 *data);
 
+/* QMTM Error handling */
+int xgene_qmtm_enable_error(struct xgene_qmtm *qmtm);
+void xgene_qmtm_disable_error(struct xgene_qmtm *qmtm);
+
 struct xgene_qmtm_sdev *storm_qmtm_get_sdev(char *name);
 
 #endif /* __XGENE_QMTM_MAIN_H__ */
Signature · 1 line
1.7.9.5
maintainer
#9 lore raw
In reply to Ravi Patel
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Arnd

#10 lore raw

Hi Arnd,

Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

-Loc

maintainer
#11 lore raw
Quoted 15 lines · On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

Arnd

author
#12 lore raw
Quoted 25 lines · Arnd Bergmann
Quoted 15 lines · On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Quoted 8 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

For simplicity, I have shown only Ethernet.
PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

maintainer
#13 lore raw
Quoted 33 lines · Ravi Patel
Quoted 32 lines · Arnd Bergmann
Quoted 15 lines · On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Quoted 6 lines
Quoted 1 line

Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

There is nothing at this link :(

Quoted 1 line

PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

How does a "security" subsystem have anything to do with ethernet?

confused,

greg k-h

author
#14 lore raw
Quoted 34 lines · Greg KH
Quoted 33 lines · Ravi Patel
Quoted 32 lines · Arnd Bergmann
Quoted 15 lines · On Saturday 21 December 2013 17:00:51 Loc Ho wrote:
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

There is nothing at this link :(

The URL is pointing to text document.
I created another URL to the same document in pdf format.
https://docs.google.com/file/d/0B28TgQZ3JLoRbXJiUXJjSUNxeDA/edit?usp=sharing&pli=1

Quoted 5 lines
Quoted 1 line

PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

How does a "security" subsystem have anything to do with ethernet?

confused,

PktDMA and Security are completely independent of Ethernet. Security Subsystem (SEC) is a CPU offload packet processing engine for security
applications.
PktDMA subsystem is a general purpose data transfer engine which moves data
from one location in memory to another.
PktDMA can perform certain data manipulation functions like XOR as data is passing
through the engine.
Both PktDMA and Security engine has their own QMTM work message format which
will be prepared by their driver

#15 lore raw
In reply to Greg KH

Hi,

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

There is nothing at this link :(

I tried click on the txt file link from web site
http://www.spinics.net/lists/netdev/msg264628.html and don't see any problem with accessing the txt file.

-Loc

maintainer
#16 lore raw
Quoted 36 lines · Loc Ho

Hi,

Quoted 34 lines
Quoted 34 lines
Quoted 22 lines
Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Quoted 8 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

There is nothing at this link :(

I tried click on the txt file link from web site
http://www.spinics.net/lists/netdev/msg264628.html and don't see any problem with accessing the txt file.

Now it's working :)

Anyway, as this is a .txt file, why wasn't it just sent in email, and added to the patch series as a documentation file?

thanks,

greg k-h

author
#17 lore raw
Quoted 41 lines · Greg KH
Quoted 40 lines · Loc Ho

Hi,

Quoted 37 lines
Quoted 29 lines
Quoted 29 lines
Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

Quoted 7 lines

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code.

In particular, please describe how this differs from a dmaengine driver and why it is not possible to extend the dma slave API to describe qmtm as a dma engine.

Quoted 8 lines

[Loc Ho]
If the QM driver implements the DMA API, what about the actual DMA engine driver which interfaces with this QM driver. We would have DMA client interfaces with the X-Gene DMA driver (not available yet) via DMA API which in turn interfaces with this QM driver via DMA API. Won't this be kind of awkward? Also, the QM only manage messages (or descriptors) which are 32-bytes or 64-bytes. It doesn't actually do any data transfer of various sizes.

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Quoted 6 lines
Quoted 1 line

Please provide a link to the data sheet if you are unable to explain.

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

There is nothing at this link :(

Quoted 8 lines
Quoted 3 lines

I tried click on the txt file link from web site
http://www.spinics.net/lists/netdev/msg264628.html and don't see any problem with accessing the txt file.

Now it's working :)

Anyway, as this is a .txt file, why wasn't it just sent in email, and added to the patch series as a documentation file?

Sure I will include the text documentation to an appropriate place in the Documentation directory in my next version of QMTM patch.

maintainer
#18 lore raw
In reply to Ravi Patel
Quoted 13 lines · Ravi Patel
Quoted 11 lines · Arnd Bergmann
Quoted 1 line · On Saturday 21 December 2013 17:00:51 Loc Ho wrote:

On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <[email protected]> wrote:

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

Quoted 8 lines

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

For simplicity, I have shown only Ethernet.
PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

Thanks, that helps a lot. Please add this file to an appropriate place in the Documentation directory in the next version of your patches.

There is still one central aspect that remains unclear to me, which is what the QMTM is actually good for, as opposed to how it gets used from the OS. In the text description, it sounds like the ethernet is the DMA master and performs DMA all by itself but from that it's not clear why a message to and from the QMTM is needed. From your drawing on the other hand, it seems like the QMTM is really the DMA master and performs the DMA on behalf of the ethernet device, which isn't connected to the coherent interface itself. If this is correct, it seems that QMTM is more like a DMA engine after all that should use the existing slave API to provide services to slave drivers.

Arnd

author
#19 lore raw
Quoted 22 lines · Arnd Bergmann
Quoted 13 lines · Ravi Patel
Quoted 11 lines · Arnd Bergmann
Quoted 1 line · On Saturday 21 December 2013 17:00:51 Loc Ho wrote:

On Sat, Dec 21, 2013 at 12:11 PM, Arnd Bergmann <[email protected]> wrote:

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

Quoted 7 lines

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

For simplicity, I have shown only Ethernet.
PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

Quoted 6 lines

Thanks, that helps a lot. Please add this file to an appropriate place in the Documentation directory in the next version of your patches.

There is still one central aspect that remains unclear to me, which is what the QMTM is actually good for, as opposed to how it gets used from the OS.

The document shows the run-time flow of messages between CPU, QMTM and Ethernet. QMTM is a centralized resource manager/driver which exports APIs to
1. Initialize & allocate queue & pbn to the Ethernet, PktDMA and Security subsystem.
2. Read queue state so that subsystems driver knows how much more work it can offload
to its subsystem.
3. Apply QoS for subsystems on their queues.

Quoted 8 lines

In the text description, it sounds like the ethernet is the DMA master and performs DMA all by itself but from that it's not clear why a message to and from the QMTM is needed. From your drawing on the other hand, it seems like the QMTM is really the DMA master and performs the DMA on behalf of the ethernet device, which isn't connected to the coherent interface itself. If this is correct, it seems that QMTM is more like a DMA engine after all that should use the existing slave API to provide services to slave drivers.

Each subsystem defines their own format of work message.
A message (or queue descriptor) has attribute fields (QMTM specific) which is common
for all subsystem work messages.
The remaining fields of a message are specific to subsystem. QMTM device doesn't have any knowledge of these subsystem specific fields and the data
operation which subsystem is going to perform using these fields. e.g.
1. Ethernet work message includes data address & length which is used by Ethernet
DMA engine for copying the data to/from its internal FIFO
2. PktDMA work message includes multiple data addresses & lengths for doing scatter/gather,
XOR operations and result data address/es to give back result to the CPU (driver)
3. Security work message includes data address & length for doing encryption or decryption,
the type of encryption or decryption and result data address to give back result to the CPU (driver)

author
#20 lore raw
Quoted 58 lines · Ravi Patel
Quoted 20 lines · Arnd Bergmann
Quoted 11 lines · Ravi Patel
Quoted 9 lines · Arnd Bergmann

Please describe here what the purpose of the qmtm is, as this is not entirely clear from the code or from your reply.

Greg was guessing that it's a bus controller, my best guess is a DMA engine. If it's something completely different, you have to let us know what it is so we can do a proper review rather than guessing.

Please provide a link to the data sheet if you are unable to explain.

Quoted 7 lines

Here is URL to a text document explaining role of QMTM device with CPU, Ethernet subsystem.

https://drive.google.com/file/d/0B28TgQZ3JLoRRGNnbjJoUGNHWW8/edit?usp=sharing

For simplicity, I have shown only Ethernet.
PktDMA and Security subsystem interfaces with QMTM in the same way as Ethernet.

Quoted 6 lines

Thanks, that helps a lot. Please add this file to an appropriate place in the Documentation directory in the next version of your patches.

There is still one central aspect that remains unclear to me, which is what the QMTM is actually good for, as opposed to how it gets used from the OS.

The document shows the run-time flow of messages between CPU, QMTM and Ethernet. QMTM is a centralized resource manager/driver which exports APIs to
1. Initialize & allocate queue & pbn to the Ethernet, PktDMA and Security subsystem.
2. Read queue state so that subsystems driver knows how much more work it can offload
to its subsystem.
3. Apply QoS for subsystems on their queues.

Quoted 8 lines

In the text description, it sounds like the ethernet is the DMA master and performs DMA all by itself but from that it's not clear why a message to and from the QMTM is needed. From your drawing on the other hand, it seems like the QMTM is really the DMA master and performs the DMA on behalf of the ethernet device, which isn't connected to the coherent interface itself. If this is correct, it seems that QMTM is more like a DMA engine after all that should use the existing slave API to provide services to slave drivers.

Each subsystem defines their own format of work message.
A message (or queue descriptor) has attribute fields (QMTM specific) which is common
for all subsystem work messages.
The remaining fields of a message are specific to subsystem. QMTM device doesn't have any knowledge of these subsystem specific fields and the data
operation which subsystem is going to perform using these fields. e.g.
1. Ethernet work message includes data address & length which is used by Ethernet
DMA engine for copying the data to/from its internal FIFO

Quoted 8 lines

2. PktDMA work message includes multiple data addresses & lengths for doing scatter/gather,
XOR operations and result data address/es to give back result to the CPU (driver)
3. Security work message includes data address & length for doing encryption or decryption,
the type of encryption or decryption and result data address to give back result to the CPU (driver)

Do you want any further clarification or document related to QMTM. We want to make sure everyone is on same page, understand and conclude upon that QMTM is a device and and not a bus or a dma engine.

maintainer
#21 lore raw
Quoted 4 lines · Ravi Patel

Do you want any further clarification or document related to QMTM. We want to make sure everyone is on same page, understand and conclude upon that QMTM is a device and and not a bus or a dma engine.

I have a much better understanding now, but there are still a few open questions from my side. Let me try to explain in my own words what I think is the relevant information (part of this is still guessing). It took me a while to figure out what it does from your description, and then some more time to see what it's actually good for (as opposed to adding complexity).

Please confirm or correct the individual statements in this description:

The QMTM serves as a relay for short (a few bytes) messages between the OS software and various slave hardware blocks on the SoC. The messages are typically but not always DMA descriptors used by the slave device used for starting bus master transactions by the slave, or for notifying sofware about the competion of a DMA transaction.

The message format is specific to the slave device and the QMTM only understands the common header of the message.

OS software sees the messages in cache-coherent memory and does not require any cache flushes or MMIO access for inbound messages and only a single posted MMIO write for outbound messages.

The queues are likely designed to be per-thread and don't require software-side locking.

For outbound messages, the QMTM is the bus master of a device-to-device DMA transaction that gets started once a message is queued and the device has signaled that it is ready for receiving it. The QMTM needs to know the bus address of the device as well as a slave ID for the signal pin.
For inbound messages, the QMTM slave initiates a busmaster transaction and needs to know the bus address of its QMTM port, while the QMTM needs to know only the slave ID that is associated with the queue.

In addition to those hardware properties, the QMTM driver needs to set up a memory buffer for the message queue as seen by the CPU, and needs tell the QMTM the location as well as some other properties such as the message length.

For inbound messages, the QMTM serves a similar purpose as an MSI controller, ensuring that inbound DMA data has arrived in RAM before an interrupt is delivered to the CPU and thereby avoiding the need for an expensive MMIO read to serialize the DMA.

The resources managed by the QMTM are both SoC-global (e.g. bus bandwidth) and slave specific (e.g. ethernet bandwith or buffer space). Global resource management is performed to prevent one slave device from monopolizing the system or preventing other slaves from making forward progress.
Examples for local resource management (I had to think about this a long time, but probably some of these are wrong) would be
* balancing between multiple non-busmaster devices connected to a dma-engine
* distributing incoming ethernet data to the available CPUs based on a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag or even individual TCP connection depending on the NIC's capabilities.
* 802.1p flow control for incoming ethernet data based on the amount of data queued up between the MAC and the driver
* interrupt mitigation for both inbound data and outbound completion, by delaying the IRQ to the OS until multiple messages have arrived or a queue specific amount of time has passed.
* controlling the amount of outbound buffer space per flow to minimize buffer-bloat between an ethernet driver and the NIC hardware.
* reordering data from outbound flows based on priority.

This is basically my current interpretation, I hope I got at least some of it right this time ;-)

Arnd

author
#22 lore raw
Quoted 42 lines · Arnd Bergmann
Quoted 4 lines · Ravi Patel

Do you want any further clarification or document related to QMTM. We want to make sure everyone is on same page, understand and conclude upon that QMTM is a device and and not a bus or a dma engine.

I have a much better understanding now, but there are still a few open questions from my side. Let me try to explain in my own words what I think is the relevant information (part of this is still guessing). It took me a while to figure out what it does from your description, and then some more time to see what it's actually good for (as opposed to adding complexity).

Please confirm or correct the individual statements in this description:

The QMTM serves as a relay for short (a few bytes) messages between the OS software and various slave hardware blocks on the SoC. The messages are typically but not always DMA descriptors used by the slave device used for starting bus master transactions by the slave, or for notifying sofware about the competion of a DMA transaction.

The message format is specific to the slave device and the QMTM only understands the common header of the message.

OS software sees the messages in cache-coherent memory and does not require any cache flushes or MMIO access for inbound messages and only a single posted MMIO write for outbound messages.

The queues are likely designed to be per-thread and don't require software-side locking.

For outbound messages, the QMTM is the bus master of a device-to-device DMA transaction that gets started once a message is queued and the device has signaled that it is ready for receiving it. The QMTM needs to know the bus address of the device as well as a slave ID for the signal pin.
For inbound messages, the QMTM slave initiates a busmaster transaction and needs to know the bus address of its QMTM port, while the QMTM needs to know only the slave ID that is associated with the queue.

Quoted 4 lines

In addition to those hardware properties, the QMTM driver needs to set up a memory buffer for the message queue as seen by the CPU, and needs tell the QMTM the location as well as some other properties such as the message length.

Your description is correct upto this point.

Quoted 4 lines

For inbound messages, the QMTM serves a similar purpose as an MSI controller, ensuring that inbound DMA data has arrived in RAM before an interrupt is delivered to the CPU and thereby avoiding the need for an expensive MMIO read to serialize the DMA.

For inbound messages, slave device generates message on a completion of a inbound DMA operation or any relevant operation targeted to the CPU. The QMTM's role is to just trigger an interrupt to CPU when there is a new message arrived from a slave device. QMTM doesn't know what the message was for. It is upto the upper layer drivers to decide how to process this message.

Quoted 21 lines

The resources managed by the QMTM are both SoC-global (e.g. bus bandwidth) and slave specific (e.g. ethernet bandwith or buffer space). Global resource management is performed to prevent one slave device from monopolizing the system or preventing other slaves from making forward progress.
Examples for local resource management (I had to think about this a long time, but probably some of these are wrong) would be
* balancing between multiple non-busmaster devices connected to a dma-engine
* distributing incoming ethernet data to the available CPUs based on a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag or even individual TCP connection depending on the NIC's capabilities.
* 802.1p flow control for incoming ethernet data based on the amount of data queued up between the MAC and the driver
* interrupt mitigation for both inbound data and outbound completion, by delaying the IRQ to the OS until multiple messages have arrived or a queue specific amount of time has passed.
* controlling the amount of outbound buffer space per flow to minimize buffer-bloat between an ethernet driver and the NIC hardware.
* reordering data from outbound flows based on priority.

Quoted 2 lines

This is basically my current interpretation, I hope I got at least some of it right this time ;-)

You have got them right. Although we have taken Ethernet examples here, most of the local resource management apply to other slave devices also.

maintainer
#23 lore raw
Quoted 11 lines · Ravi Patel
Quoted 4 lines

For inbound messages, the QMTM serves a similar purpose as an MSI controller, ensuring that inbound DMA data has arrived in RAM before an interrupt is delivered to the CPU and thereby avoiding the need for an expensive MMIO read to serialize the DMA.

For inbound messages, slave device generates message on a completion of a inbound DMA operation or any relevant operation targeted to the CPU. The QMTM's role is to just trigger an interrupt to CPU when there is a new message arrived from a slave device. QMTM doesn't know what the message was for. It is upto the upper layer drivers to decide how to process this message.

That doesn't seem to contradict what I wrote above. The DMA ordering would be an implicit side-effect of the message generated by the slave device if the QMTM is on the same bus as the external memory controller and the message has the "strict ordering" bit set on the bus transaction.

Arnd

maintainer
#24 lore raw
In reply to Ravi Patel
Quoted 16 lines · Ravi Patel
Quoted 16 lines

Examples for local resource management (I had to think about this a long time, but probably some of these are wrong) would be
* balancing between multiple non-busmaster devices connected to a dma-engine
* distributing incoming ethernet data to the available CPUs based on a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag or even individual TCP connection depending on the NIC's capabilities.
* 802.1p flow control for incoming ethernet data based on the amount of data queued up between the MAC and the driver
* interrupt mitigation for both inbound data and outbound completion, by delaying the IRQ to the OS until multiple messages have arrived or a queue specific amount of time has passed.
* controlling the amount of outbound buffer space per flow to minimize buffer-bloat between an ethernet driver and the NIC hardware.
* reordering data from outbound flows based on priority.

Quoted 5 lines
Quoted 2 lines

This is basically my current interpretation, I hope I got at least some of it right this time ;-)

You have got them right. Although we have taken Ethernet examples here, most of the local resource management apply to other slave devices also.

I'm very suprised I got all those right, it seems it's a quite sophisticated piece of hardware then. I guess most other slave devices only use a subset of the capabilities that ethernet wants.

Now that I have a better understanding of what the driver is good for and how it's used, let's have a look at how we can make it fit into the Linux driver APIs and the DT bindings. We don't have anything exactly like this yet, but I think the "mailbox" framework is a close enough match that we can fit it in there, with some extensions. This framework is still in the process of being created (so far there is only a TI OMAP specific driver, and one for pl320), and I've not seen any mailboxes that support IRQ mitigation or multiple software interfaces per hardware mailbox, but those should be easy enough to add.

For the DT binding, I would suggest using something along the lines of what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this (yet), but now would be a good time to standardize it. The QMTM node should define a "#mailbox-cells" property that indicates how many 32-bit cells a qmtm needs to describe the connection between the controller and the slave. My best guess is that this would be hardcoded to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell for the slave-id signal number. All other parameters that you have in the big table in the qmtm driver at the moment can then get moved into the slave drivers, as they are constant per type of slave. This will simplify the QMTM driver.

In the slave, you should have a "mailboxes" property with a phandle to the qmtm followed by the three cells to identify the actual queue. If it's possible that a device uses more than one rx and one tx queue, we also need a "mailbox-names" property to identify the individual queues.

For the in-kernel interfaces, we should probably start a conversation with the owners of the mailbox drivers to get a common API, for now I'd suggest you just leave it as it is, and only adapt for the new binding.

Arnd

author
#25 lore raw
Quoted 9 lines · Arnd Bergmann

Now that I have a better understanding of what the driver is good for and how it's used, let's have a look at how we can make it fit into the Linux driver APIs and the DT bindings. We don't have anything exactly like this yet, but I think the "mailbox" framework is a close enough match that we can fit it in there, with some extensions. This framework is still in the process of being created (so far there is only a TI OMAP specific driver, and one for pl320), and I've not seen any mailboxes that support IRQ mitigation or multiple software interfaces per hardware mailbox, but those should be easy enough to add.

OK. We will put Queue Manager driver under drivers/mailbox directory along with TI OMAP and pl320 drivers.

Quoted 12 lines

For the DT binding, I would suggest using something along the lines of what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this (yet), but now would be a good time to standardize it. The QMTM node should define a "#mailbox-cells" property that indicates how many 32-bit cells a qmtm needs to describe the connection between the controller and the slave. My best guess is that this would be hardcoded to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell for the slave-id signal number. All other parameters that you have in the big table in the qmtm driver at the moment can then get moved into the slave drivers, as they are constant per type of slave. This will simplify the QMTM driver.

Quoted 5 lines

In the slave, you should have a "mailboxes" property with a phandle to the qmtm followed by the three cells to identify the actual queue. If it's possible that a device uses more than one rx and one tx queue, we also need a "mailbox-names" property to identify the individual queues.

We explored on DT bindings suggestion given by you. We have come up with a sample DT binding for how it will look like. Herewith we have provided the same. Would you please review and give us your comments before we change our driver and DTS file to accomodate it?

Sample DTS node for QM:

                qmlite: qmtm at 17030000 {
                        compatible = "apm,xgene-qmtm-lite";
                        reg = <0x0 0x17030000 0x0 0x10000>,
                              <0x0 0x10000000 0x0 0x400000>;
                        interrupts = <0x0 0x40 0x4>,
                                     <0x0 0x3c 0x4>;
                        status = "ok";
                        #clock-cells = <1>;
                        clocks = <&qmlclk 0>;
                        #mailbox-cells = <3>;
                };

Sample DTS node for Ethernet:

                menet: ethernet at 17020000 {
                        compatible = "apm,xgene-enet";
                        status = "disabled";
                        reg = <0x0 0x17020000 0x0 0x30>,
                              <0x0 0x17020000 0x0 0x10000>,
                              <0x0 0x17020000 0x0 0x20>;
                        mailboxes = <&qmlite 0x0 0x1000002c 0x0000>,
                                            <&qmlite 0x0 0x10000052 0x0020>,
                                            <&qmlite 0x0 0x10000060 0x0f00>
                        mailbox-names = "mb-tx", "mb-fp", "mb-rx";
                        interrupts = <0x0 0x38 0x4>,
                                     <0x0 0x39 0x4>,
                                     <0x0 0x3a 0x4>;
                        #clock-cells = <1>;
                        clocks = <&eth8clk 0>;
                        local-mac-address = <0x0 0x11 0x3a 0x8a 0x5a 0x78>;
                        max-frame-size = <0x233a>;
                        phyid = <3>;
                        phy-mode = "rgmii";
                };

The mailbox node in DTS has following format:
mailbox = <&parent 'higher 32 bit bus address' 'lower 32 bit bus address' 'signal id'>
Ethernet driver will call following function in platform_probe:
mailbox_get(dev, "mb-tx");
mailbox_get API will return the the context of allocated and configured mailbox. For now, mailbox_get API will be implemented in xgene QMTM driver. Eventually when mailbox framework in Linux will be standardized, we will use it instead.

Quoted 4 lines

For the in-kernel interfaces, we should probably start a conversation with the owners of the mailbox drivers to get a common API, for now I'd suggest you just leave it as it is, and only adapt for the new binding.

Sure. For now we will put our driver mostly as is in the drivers/mailbox. Can you please help us identify the owners of the mailbox drivers? As you suggested, we can start conversation with them to define common in kernel APIs.

Ravi

Quoted 61 lines · Arnd Bergmann
Quoted 16 lines · Ravi Patel
Quoted 16 lines

Examples for local resource management (I had to think about this a long time, but probably some of these are wrong) would be
* balancing between multiple non-busmaster devices connected to a dma-engine
* distributing incoming ethernet data to the available CPUs based on a flow classifier in the MAC, e.g. by IOV MAC address, VLAN tag or even individual TCP connection depending on the NIC's capabilities.
* 802.1p flow control for incoming ethernet data based on the amount of data queued up between the MAC and the driver
* interrupt mitigation for both inbound data and outbound completion, by delaying the IRQ to the OS until multiple messages have arrived or a queue specific amount of time has passed.
* controlling the amount of outbound buffer space per flow to minimize buffer-bloat between an ethernet driver and the NIC hardware.
* reordering data from outbound flows based on priority.

Quoted 5 lines
Quoted 2 lines

This is basically my current interpretation, I hope I got at least some of it right this time ;-)

You have got them right. Although we have taken Ethernet examples here, most of the local resource management apply to other slave devices also.

I'm very suprised I got all those right, it seems it's a quite sophisticated piece of hardware then. I guess most other slave devices only use a subset of the capabilities that ethernet wants.

Now that I have a better understanding of what the driver is good for and how it's used, let's have a look at how we can make it fit into the Linux driver APIs and the DT bindings. We don't have anything exactly like this yet, but I think the "mailbox" framework is a close enough match that we can fit it in there, with some extensions. This framework is still in the process of being created (so far there is only a TI OMAP specific driver, and one for pl320), and I've not seen any mailboxes that support IRQ mitigation or multiple software interfaces per hardware mailbox, but those should be easy enough to add.

For the DT binding, I would suggest using something along the lines of what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this (yet), but now would be a good time to standardize it. The QMTM node should define a "#mailbox-cells" property that indicates how many 32-bit cells a qmtm needs to describe the connection between the controller and the slave. My best guess is that this would be hardcoded to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell for the slave-id signal number. All other parameters that you have in the big table in the qmtm driver at the moment can then get moved into the slave drivers, as they are constant per type of slave. This will simplify the QMTM driver.

In the slave, you should have a "mailboxes" property with a phandle to the qmtm followed by the three cells to identify the actual queue. If it's possible that a device uses more than one rx and one tx queue, we also need a "mailbox-names" property to identify the individual queues.

For the in-kernel interfaces, we should probably start a conversation with the owners of the mailbox drivers to get a common API, for now I'd suggest you just leave it as it is, and only adapt for the new binding.

Arnd

maintainer
#26 lore raw
Quoted 1 line · Ravi Patel

On Tue, Jan 14, 2014 at 7:15 AM, Arnd Bergmann <[email protected]> wrote:

-

Quoted 18 lines
Quoted 12 lines

For the DT binding, I would suggest using something along the lines of what we have for clocks, pinctrl and dmaengine. OMAP doesn't use this (yet), but now would be a good time to standardize it. The QMTM node should define a "#mailbox-cells" property that indicates how many 32-bit cells a qmtm needs to describe the connection between the controller and the slave. My best guess is that this would be hardcoded to <3>, using two cells for a 64-bit FIFO bus address, and a 32-bit cell for the slave-id signal number. All other parameters that you have in the big table in the qmtm driver at the moment can then get moved into the slave drivers, as they are constant per type of slave. This will simplify the QMTM driver.

Quoted 5 lines

In the slave, you should have a "mailboxes" property with a phandle to the qmtm followed by the three cells to identify the actual queue. If it's possible that a device uses more than one rx and one tx queue, we also need a "mailbox-names" property to identify the individual queues.

Quoted 8 lines

We explored on DT bindings suggestion given by you. We have come up with a sample DT binding for how it will look like. Herewith we have provided the same. Would you please review and give us your comments before we change our driver and DTS file to accomodate it?

Sample DTS node for QM:

                qmlite: qmtm at 17030000 {
                        compatible = "apm,xgene-qmtm-lite";

I would use 'mailbox at 17030000' as the node name, as the name part is supposed to be descriptive of the function rather than the implemention.

Quoted 9 lines
                        reg = <0x0 0x17030000 0x0 0x10000>,
                              <0x0 0x10000000 0x0 0x400000>;
                        interrupts = <0x0 0x40 0x4>,
                                     <0x0 0x3c 0x4>;
                        status = "ok";
                        #clock-cells = <1>;
                        clocks = <&qmlclk 0>;
                        #mailbox-cells = <3>;
                };

The #clock-cells seems misplaced here, unless this is also a clock provider, which I don't think it is.

Quoted 8 lines

Sample DTS node for Ethernet:

                menet: ethernet at 17020000 {
                        compatible = "apm,xgene-enet";
                        status = "disabled";
                        reg = <0x0 0x17020000 0x0 0x30>,
                              <0x0 0x17020000 0x0 0x10000>,
                              <0x0 0x17020000 0x0 0x20>;

Unrelated, but it seems strange to have three register sets of different sizes at the same offset.

Quoted 4 lines
                        mailboxes = <&qmlite 0x0 0x1000002c 0x0000>,
                                            <&qmlite 0x0 0x10000052 0x0020>,
                                            <&qmlite 0x0 0x10000060 0x0f00>
                        mailbox-names = "mb-tx", "mb-fp", "mb-rx";

I would leave out the "mb-" part of the strings and just document them as "tx", "rx" and "fp".

Quoted 4 lines
                        interrupts = <0x0 0x38 0x4>,
                                     <0x0 0x39 0x4>,
                                     <0x0 0x3a 0x4>;
                        #clock-cells = <1>;

Same comment about #clock-cells here.

Quoted 10 lines
                        clocks = <&eth8clk 0>;
                        local-mac-address = <0x0 0x11 0x3a 0x8a 0x5a 0x78>;
                        max-frame-size = <0x233a>;
                        phyid = <3>;
                        phy-mode = "rgmii";
                };

The mailbox node in DTS has following format:
mailbox = <&parent 'higher 32 bit bus address' 'lower 32 bit bus address' 'signal id'>

sounds good.

Quoted 6 lines

Ethernet driver will call following function in platform_probe:
mailbox_get(dev, "mb-tx");
mailbox_get API will return the the context of allocated and configured mailbox. For now, mailbox_get API will be implemented in xgene QMTM driver. Eventually when mailbox framework in Linux will be standardized, we will use it instead.

Ok.

Quoted 9 lines
Quoted 4 lines

For the in-kernel interfaces, we should probably start a conversation with the owners of the mailbox drivers to get a common API, for now I'd suggest you just leave it as it is, and only adapt for the new binding.

Sure. For now we will put our driver mostly as is in the drivers/mailbox. Can you please help us identify the owners of the mailbox drivers? As you suggested, we can start conversation with them to define common in kernel APIs.

Please talk to "Suman Anna" <[email protected]> for the TI part and Rob Herring <[email protected]> for pl320. The pl320 driver was written by Mark Langsdorf for Calxeda, but I don't have an updated email address for him and assume that the calxeda address is no longer functional.

Arnd

maintainer
#27 lore raw
In reply to Ravi Patel
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

As you failed to respond or even acknowledge any of my review comments before, I don't understand what you expect me to do with this patch dump again.

Remember, kernel development is about trust. You need me to trust that you know what you are doing, and will be able to handle feedback and respond to issues that come up over time. Right now that trust is in the negative zone, around "general annoyance" levels. It's up to you to change that or not, just like it is up to me to take your code, or not.

grumble,

greg k-h

author
#28 lore raw
Quoted 17 lines · Greg KH
Quoted 5 lines · Ravi Patel

This patch adds support for APM X-Gene SoC Queue Manager/Traffic Manager.
QMTM is required by APM X-Gene SoC Ethernet, PktDMA (XOR Engine) and Security Engine subsystems. All subsystems communicate with QMTM using messages which include information about the work to be performed and the location of associated data buffers.

As you failed to respond or even acknowledge any of my review comments before, I don't understand what you expect me to do with this patch dump again.

Remember, kernel development is about trust. You need me to trust that you know what you are doing, and will be able to handle feedback and respond to issues that come up over time. Right now that trust is in the negative zone, around "general annoyance" levels. It's up to you to change that or not, just like it is up to me to take your code, or not.

Quoted 3 lines

grumble,

greg k-h

Greg,

I apologize for not responding to your comments before sending patch v2 The main intention of sending patch v2 was to remove APM confidential notice, which was present in the first QMTM patch.
I wanted to send out patch v2 quickly, so that community can start reviewing
the QMTM code.

I will go through all your comments and respond it.

pardon,

Ravi