Skip to contentSkip to conversation

perf x86: Exposing IO stack to IO PMON mapping through sysfs

PATCH Superseded [email protected] 2 participants 5 messages Dec 12, 2019 – Dec 19, 2019

Subsystems (2)

v3 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v3 cover perf x86: Exposing IO stack to IO PMON mapping through sysfs

From: Roman Sudarikov <[email protected]>

The previous version can be found at:
v2: https://lkml.org/lkml/2019/12/10/185

Changes in this revision are:
v2 -> v3:
- Addressed comments from Peter and Kan

The previous version can be found at:
v1: https://lkml.org/lkml/2019/11/26/447

Changes in this revision are:
v1 -> v2:
- Fixed process related issues;
- This patch set includes kernel support for IIO stack to PMON mapping;
- Stephane raised concerns regarding output format which may require code changes in the user space part of the feature only. We will continue output format discussion in the context of user space update.

Intel® Xeon® Scalable processor family (code name Skylake-SP) makes significant changes in the integrated I/O (IIO) architecture. The new solution introduces IIO stacks which are responsible for managing traffic between the PCIe domain and the Mesh domain. Each IIO stack has its own PMON block and can handle either DMI port, x16 PCIe root port, MCP-Link or various built-in accelerators. IIO PMON blocks allow concurrent monitoring of I/O flows up to 4 x4 bifurcation within each IIO stack.

Software is supposed to program required perf counters within each IIO stack and gather performance data. The tricky thing here is that IIO PMON reports data per IIO stack but users have no idea what IIO stacks are - they only know devices which are connected to the platform.

Understanding IIO stack concept to find which IIO stack that particular IO device is connected to, or to identify an IIO PMON block to program for monitoring specific IIO stack assumes a lot of implicit knowledge about given Intel server platform architecture.

This patch set introduces:
1. An infrastructure for exposing an Uncore unit to Uncore PMON mapping through sysfs-backend;
2. A new --iiostat mode in perf stat to provide I/O performance metrics per I/O device.

Usage examples:

1. List all devices below IIO stacks
./perf stat --iiostat=show

Sample output w/o libpci:

    S0-RootPort0-uncore_iio_0<00:00.0>
    S1-RootPort0-uncore_iio_0<81:00.0>
    S0-RootPort1-uncore_iio_1<18:00.0>
    S1-RootPort1-uncore_iio_1<86:00.0>
    S1-RootPort1-uncore_iio_1<88:00.0>
    S0-RootPort2-uncore_iio_2<3d:00.0>
    S1-RootPort2-uncore_iio_2<af:00.0>
    S1-RootPort3-uncore_iio_3<da:00.0>

Sample output with libpci:

    S0-RootPort0-uncore_iio_0<00:00.0 Sky Lake-E DMI3 Registers>
    S1-RootPort0-uncore_iio_0<81:00.0 Ethernet Controller X710 for 10GbE SFP+>
    S0-RootPort1-uncore_iio_1<18:00.0 Omni-Path HFI Silicon 100 Series [discrete]>
    S1-RootPort1-uncore_iio_1<86:00.0 Ethernet Controller XL710 for 40GbE QSFP+>
    S1-RootPort1-uncore_iio_1<88:00.0 Ethernet Controller XL710 for 40GbE QSFP+>
    S0-RootPort2-uncore_iio_2<3d:00.0 Ethernet Connection X722 for 10GBASE-T>
    S1-RootPort2-uncore_iio_2<af:00.0 Omni-Path HFI Silicon 100 Series [discrete]>
    S1-RootPort3-uncore_iio_3<da:00.0 NVMe Datacenter SSD [Optane]>

2. Collect metrics for all I/O devices below IIO stack

  ./perf stat --iiostat -- dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct
    357708+0 records in
    357707+0 records out
    375083606016 bytes (375 GB, 349 GiB) copied, 215.381 s, 1.7 GB/s

Performance counter stats for 'system wide':

device Inbound Read(MB) Inbound Write(MB) Outbound Read(MB) Outbound Write(MB)
00:00.0 0 0 0 0
81:00.0 0 0 0 0
18:00.0 0 0 0 0
86:00.0 0 0 0 0
88:00.0 0 0 0 0
3b:00.0 3 0 0 0
3c:03.0 3 0 0 0
3d:00.0 3 0 0 0
af:00.0 0 0 0 0
da:00.0 358559 44 0 22

215.383783574 seconds time elapsed

3. Collect metrics for comma separted list of I/O devices

  ./perf stat --iiostat=da:00.0 -- dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct
    381555+0 records in
    381554+0 records out
    400088457216 bytes (400 GB, 373 GiB) copied, 374.044 s, 1.1 GB/s

Performance counter stats for 'system wide':

device Inbound Read(MB) Inbound Write(MB) Outbound Read(MB) Outbound Write(MB)
da:00.0 382462 47 0 23

374.045775505 seconds time elapsed

Roman Sudarikov (2):
perf x86: Infrastructure for exposing an Uncore unit to PMON mapping perf x86: Exposing an Uncore unit to PMON for Intel Xeon® server
platform

 arch/x86/events/intel/uncore.c       |  39 ++++++-
 arch/x86/events/intel/uncore.h       |  10 +-
 arch/x86/events/intel/uncore_snbep.c | 162 +++++++++++++++++++++++++++
 3 files changed, 208 insertions(+), 3 deletions(-)

base-commit: 219d54332a09e8d8741c1e1982f5eae56099de85

Signature · 1 line
2.19.1
#1 lore raw

PATCH v3 1/2 perf x86: Infrastructure for exposing an Uncore unit to PMON mapping

From: Roman Sudarikov <[email protected]>

Intel® Xeon® Scalable processor family (code name Skylake-SP) makes significant changes in the integrated I/O (IIO) architecture. The new solution introduces IIO stacks which are responsible for managing traffic between the PCIe domain and the Mesh domain. Each IIO stack has its own PMON block and can handle either DMI port, x16 PCIe root port, MCP-Link or various built-in accelerators. IIO PMON blocks allow concurrent monitoring of I/O flows up to 4 x4 bifurcation within each IIO stack.

Software is supposed to program required perf counters within each IIO stack and gather performance data. The tricky thing here is that IIO PMON reports data per IIO stack but users have no idea what IIO stacks are - they only know devices which are connected to the platform.

Understanding IIO stack concept to find which IIO stack that particular IO device is connected to, or to identify an IIO PMON block to program for monitoring specific IIO stack assumes a lot of implicit knowledge about given Intel server platform architecture.

Usage example:
/sys/devices/uncore_<type>_<pmu_idx>/platform_mapping

Each Uncore unit type, by its nature, can be mapped to its own context, for example:
1. CHA - each uncore_cha_<pmu_idx> is assigned to manage a distinct slice of LLC capacity;
2. UPI - each uncore_upi_<pmu_idx> is assigned to manage one link of Intel UPI Subsystem;
3. IIO - each uncore_iio_<pmu_idx> is assigned to manage one stack of the IIO module;
4. IMC - each uncore_imc_<pmu_idx> is assigned to manage one channel of Memory Controller.

Implementation details:
Two callbacks added to struct intel_uncore_type to discover and map Uncore units to PMONs:
int (*get_topology)(void)
int (*set_mapping)(struct intel_uncore_pmu *pmu)

Details of IIO Uncore unit mapping to IIO PMON:
Each IIO stack is either DMI port, x16 PCIe root port, MCP-Link or various built-in accelerators. For Uncore IIO Unit type, the platform_mapping file holds bus numbers of devices, which can be monitored by that IIO PMON block on each die.

---
 arch/x86/events/intel/uncore.c | 37 +++++++++++++++++++++++++++++++++-
 arch/x86/events/intel/uncore.h |  9 ++++++++-
 2 files changed, 44 insertions(+), 2 deletions(-)
arch/x86/events/intel/uncore.c +36 -1
index 86467f85c383..2c53ad44b51f 100644
@@ -905,6 +905,32 @@ static void uncore_types_exit(struct intel_uncore_type **types)
 		uncore_type_exit(*types);
 }
 
+static struct attribute *empty_attrs[] = {
+	NULL,
+};
+
+static const struct attribute_group empty_group = {
+	.attrs = empty_attrs,
+};
+
+static ssize_t platform_mapping_show(struct device *dev,
+				struct device_attribute *attr, char *buf)
+{
+	struct intel_uncore_pmu *pmu = dev_get_drvdata(dev);
+
+	return snprintf(buf, PAGE_SIZE - 1, "%s\n", pmu->mapping);
+}
+static DEVICE_ATTR_RO(platform_mapping);
+
+static struct attribute *mapping_attrs[] = {
+	&dev_attr_platform_mapping.attr,
+	NULL,
+};
+
+static const struct attribute_group uncore_mapping_group = {
+	.attrs = mapping_attrs,
+};
+
 static int __init uncore_type_init(struct intel_uncore_type *type, bool setid)
 {
 	struct intel_uncore_pmu *pmus;
@@ -950,10 +976,19 @@ static int __init uncore_type_init(struct intel_uncore_type *type, bool setid)
 			attr_group->attrs[j] = &type->event_descs[j].attr.attr;
 
 		type->events_group = &attr_group->group;
-	}
+	} else
+		type->events_group = &empty_group;
 
 	type->pmu_group = &uncore_pmu_attr_group;
 
+	/*
+	 * Exposing mapping of Uncore units to corresponding Uncore PMUs
+	 * through /sys/devices/uncore_<type>_<idx>/platform_mapping
+	 */
+	if (type->get_topology && type->set_mapping)
+		if (!type->get_topology(type) && !type->set_mapping(type))
+			type->mapping_group = &uncore_mapping_group;
+
 	return 0;
 
 err:
arch/x86/events/intel/uncore.h +8 -1
index bbfdaa720b45..f52dd3f112a7 100644
@@ -71,13 +71,19 @@ struct intel_uncore_type {
 	struct intel_uncore_ops *ops;
 	struct uncore_event_desc *event_descs;
 	struct freerunning_counters *freerunning;
-	const struct attribute_group *attr_groups[4];
+	const struct attribute_group *attr_groups[5];
 	struct pmu *pmu; /* for custom pmu ops */
+	u64 *topology;
+	/* finding Uncore units */
+	int (*get_topology)(struct intel_uncore_type *type);
+	/* mapping Uncore units to PMON ranges */
+	int (*set_mapping)(struct intel_uncore_type *type);
 };
 
 #define pmu_group attr_groups[0]
 #define format_group attr_groups[1]
 #define events_group attr_groups[2]
+#define mapping_group attr_groups[3]
 
 struct intel_uncore_ops {
 	void (*init_box)(struct intel_uncore_box *);
@@ -99,6 +105,7 @@ struct intel_uncore_pmu {
 	int				pmu_idx;
 	int				func_id;
 	bool				registered;
+	char				*mapping;
 	atomic_t			activeboxes;
 	struct intel_uncore_type	*type;
 	struct intel_uncore_box		**boxes;
Signature · 1 line
2.19.1
#2 lore raw

PATCH v3 2/2 perf x86: Exposing an Uncore unit to PMON for Intel Xeon® server platform

In reply to [email protected]

From: Roman Sudarikov <[email protected]>

Current version supports a server line starting Intel® Xeon® Processor Scalable Family and introduces mapping for IIO Uncore units only. Other units can be added on demand.

IIO stack to PMON mapping is exposed through:
/sys/devices/uncore_iio_<pmu_idx>/platform_mapping in the following format: domain:bus

For example, on a 4-die Intel Xeon® server platform:

    $ cat /sys/devices/uncore_iio_0/platform_mapping
    0000:00,0000:40,0000:80,0000:c0

Which means:
IIO PMON block 0 on die 0 belongs to IIO stack on bus 0x00, domain 0x0000 IIO PMON block 0 on die 1 belongs to IIO stack on bus 0x40, domain 0x0000 IIO PMON block 0 on die 2 belongs to IIO stack on bus 0x80, domain 0x0000 IIO PMON block 0 on die 3 belongs to IIO stack on bus 0xc0, domain 0x0000

---
 arch/x86/events/intel/uncore.c       |   2 +-
 arch/x86/events/intel/uncore.h       |   1 +
 arch/x86/events/intel/uncore_snbep.c | 162 +++++++++++++++++++++++++++
 3 files changed, 164 insertions(+), 1 deletion(-)
arch/x86/events/intel/uncore.c +1 -1
index 2c53ad44b51f..c0d86bc8e786 100644
@@ -16,7 +16,7 @@ struct pci_driver *uncore_pci_driver;
 DEFINE_RAW_SPINLOCK(pci2phy_map_lock);
 struct list_head pci2phy_map_head = LIST_HEAD_INIT(pci2phy_map_head);
 struct pci_extra_dev *uncore_extra_pci_dev;
-static int max_dies;
+int max_dies;
 
 /* mask of cpus that collect uncore events */
 static cpumask_t uncore_cpu_mask;
arch/x86/events/intel/uncore.h +1 -0
index f52dd3f112a7..94eacca6f485 100644
@@ -523,6 +523,7 @@ extern raw_spinlock_t pci2phy_map_lock;
 extern struct list_head pci2phy_map_head;
 extern struct pci_extra_dev *uncore_extra_pci_dev;
 extern struct event_constraint uncore_constraint_empty;
+extern int max_dies;
 
 /* uncore_snb.c */
 int snb_uncore_pci_init(void);
arch/x86/events/intel/uncore_snbep.c +162 -0

186 lines not on this page · raw message

Signature · 1 line
2.19.1
Reviewed-by on the v3 series
#3 lore raw
In reply to [email protected]
Quoted 5 lines · On 12/12/2019 10:04 AM, [email protected] wrote:

From: Roman Sudarikov <[email protected]>

The previous version can be found at:
v2: https://lkml.org/lkml/2019/12/10/185

If you have to provide next version to address other issues, please use canonical form to provide a link. For example,

https://lkml.kernel.org/r/[email protected]

"[email protected]" is the Message-ID of V2 patch set, which can help maintainers or reviewers to locate the patch set quickly.

Quoted 4 lines

Changes in this revision are:
v2 -> v3:
- Addressed comments from Peter and Kan

Please specify which comments are addressed and how are they addressed next time. So we can focus on the updated codes.

Except the change log issue, the patch set looks good to me.

Quoted 114 lines

The previous version can be found at:
v1: https://lkml.org/lkml/2019/11/26/447

Changes in this revision are:
v1 -> v2:
- Fixed process related issues;
- This patch set includes kernel support for IIO stack to PMON mapping;
- Stephane raised concerns regarding output format which may require code changes in the user space part of the feature only. We will continue output format discussion in the context of user space update.

Intel® Xeon® Scalable processor family (code name Skylake-SP) makes significant changes in the integrated I/O (IIO) architecture. The new solution introduces IIO stacks which are responsible for managing traffic between the PCIe domain and the Mesh domain. Each IIO stack has its own PMON block and can handle either DMI port, x16 PCIe root port, MCP-Link or various built-in accelerators. IIO PMON blocks allow concurrent monitoring of I/O flows up to 4 x4 bifurcation within each IIO stack.

Software is supposed to program required perf counters within each IIO stack and gather performance data. The tricky thing here is that IIO PMON reports data per IIO stack but users have no idea what IIO stacks are - they only know devices which are connected to the platform.

Understanding IIO stack concept to find which IIO stack that particular IO device is connected to, or to identify an IIO PMON block to program for monitoring specific IIO stack assumes a lot of implicit knowledge about given Intel server platform architecture.

This patch set introduces:
1. An infrastructure for exposing an Uncore unit to Uncore PMON mapping

    through sysfs-backend;

2. A new --iiostat mode in perf stat to provide I/O performance metrics per I/O device.

Usage examples:

1. List all devices below IIO stacks
./perf stat --iiostat=show

Sample output w/o libpci:

     S0-RootPort0-uncore_iio_0<00:00.0>
     S1-RootPort0-uncore_iio_0<81:00.0>
     S0-RootPort1-uncore_iio_1<18:00.0>
     S1-RootPort1-uncore_iio_1<86:00.0>
     S1-RootPort1-uncore_iio_1<88:00.0>
     S0-RootPort2-uncore_iio_2<3d:00.0>
     S1-RootPort2-uncore_iio_2<af:00.0>
     S1-RootPort3-uncore_iio_3<da:00.0>

Sample output with libpci:

     S0-RootPort0-uncore_iio_0<00:00.0 Sky Lake-E DMI3 Registers>
     S1-RootPort0-uncore_iio_0<81:00.0 Ethernet Controller X710 for 10GbE SFP+>
     S0-RootPort1-uncore_iio_1<18:00.0 Omni-Path HFI Silicon 100 Series [discrete]>
     S1-RootPort1-uncore_iio_1<86:00.0 Ethernet Controller XL710 for 40GbE QSFP+>
     S1-RootPort1-uncore_iio_1<88:00.0 Ethernet Controller XL710 for 40GbE QSFP+>
     S0-RootPort2-uncore_iio_2<3d:00.0 Ethernet Connection X722 for 10GBASE-T>
     S1-RootPort2-uncore_iio_2<af:00.0 Omni-Path HFI Silicon 100 Series [discrete]>
     S1-RootPort3-uncore_iio_3<da:00.0 NVMe Datacenter SSD [Optane]>

2. Collect metrics for all I/O devices below IIO stack

   ./perf stat --iiostat -- dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct
     357708+0 records in
     357707+0 records out
     375083606016 bytes (375 GB, 349 GiB) copied, 215.381 s, 1.7 GB/s

Performance counter stats for 'system wide':

device Inbound Read(MB) Inbound Write(MB) Outbound Read(MB) Outbound Write(MB)
00:00.0 0 0 0 0
81:00.0 0 0 0 0
18:00.0 0 0 0 0
86:00.0 0 0 0 0
88:00.0 0 0 0 0
3b:00.0 3 0 0 0
3c:03.0 3 0 0 0
3d:00.0 3 0 0 0
af:00.0 0 0 0 0
da:00.0 358559 44 0 22

215.383783574 seconds time elapsed

3. Collect metrics for comma separted list of I/O devices

   ./perf stat --iiostat=da:00.0 -- dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct
     381555+0 records in
     381554+0 records out
     400088457216 bytes (400 GB, 373 GiB) copied, 374.044 s, 1.1 GB/s

Performance counter stats for 'system wide':

device Inbound Read(MB) Inbound Write(MB) Outbound Read(MB) Outbound Write(MB)
da:00.0 382462 47 0 23

374.045775505 seconds time elapsed

Roman Sudarikov (2):
perf x86: Infrastructure for exposing an Uncore unit to PMON mapping perf x86: Exposing an Uncore unit to PMON for Intel Xeon® server
platform

  arch/x86/events/intel/uncore.c       |  39 ++++++-
  arch/x86/events/intel/uncore.h       |  10 +-
  arch/x86/events/intel/uncore_snbep.c | 162 +++++++++++++++++++++++++++
  3 files changed, 208 insertions(+), 3 deletions(-)

base-commit: 219d54332a09e8d8741c1e1982f5eae56099de85

#4 lore raw
In reply to [email protected]

Hello Peter,

could you please take a look at the patch set.

Thanks,
Roman

Quoted 122 lines · On 12.12.2019 18:04, [email protected] wrote:

From: Roman Sudarikov <[email protected]>

The previous version can be found at:
v2: https://lkml.org/lkml/2019/12/10/185

Changes in this revision are:
v2 -> v3:
- Addressed comments from Peter and Kan

The previous version can be found at:
v1: https://lkml.org/lkml/2019/11/26/447

Changes in this revision are:
v1 -> v2:
- Fixed process related issues;
- This patch set includes kernel support for IIO stack to PMON mapping;
- Stephane raised concerns regarding output format which may require code changes in the user space part of the feature only. We will continue output format discussion in the context of user space update.

Intel® Xeon® Scalable processor family (code name Skylake-SP) makes significant changes in the integrated I/O (IIO) architecture. The new solution introduces IIO stacks which are responsible for managing traffic between the PCIe domain and the Mesh domain. Each IIO stack has its own PMON block and can handle either DMI port, x16 PCIe root port, MCP-Link or various built-in accelerators. IIO PMON blocks allow concurrent monitoring of I/O flows up to 4 x4 bifurcation within each IIO stack.

Software is supposed to program required perf counters within each IIO stack and gather performance data. The tricky thing here is that IIO PMON reports data per IIO stack but users have no idea what IIO stacks are - they only know devices which are connected to the platform.

Understanding IIO stack concept to find which IIO stack that particular IO device is connected to, or to identify an IIO PMON block to program for monitoring specific IIO stack assumes a lot of implicit knowledge about given Intel server platform architecture.

This patch set introduces:
1. An infrastructure for exposing an Uncore unit to Uncore PMON mapping

    through sysfs-backend;

2. A new --iiostat mode in perf stat to provide I/O performance metrics per I/O device.

Usage examples:

1. List all devices below IIO stacks
./perf stat --iiostat=show

Sample output w/o libpci:

     S0-RootPort0-uncore_iio_0<00:00.0>
     S1-RootPort0-uncore_iio_0<81:00.0>
     S0-RootPort1-uncore_iio_1<18:00.0>
     S1-RootPort1-uncore_iio_1<86:00.0>
     S1-RootPort1-uncore_iio_1<88:00.0>
     S0-RootPort2-uncore_iio_2<3d:00.0>
     S1-RootPort2-uncore_iio_2<af:00.0>
     S1-RootPort3-uncore_iio_3<da:00.0>

Sample output with libpci:

     S0-RootPort0-uncore_iio_0<00:00.0 Sky Lake-E DMI3 Registers>
     S1-RootPort0-uncore_iio_0<81:00.0 Ethernet Controller X710 for 10GbE SFP+>
     S0-RootPort1-uncore_iio_1<18:00.0 Omni-Path HFI Silicon 100 Series [discrete]>
     S1-RootPort1-uncore_iio_1<86:00.0 Ethernet Controller XL710 for 40GbE QSFP+>
     S1-RootPort1-uncore_iio_1<88:00.0 Ethernet Controller XL710 for 40GbE QSFP+>
     S0-RootPort2-uncore_iio_2<3d:00.0 Ethernet Connection X722 for 10GBASE-T>
     S1-RootPort2-uncore_iio_2<af:00.0 Omni-Path HFI Silicon 100 Series [discrete]>
     S1-RootPort3-uncore_iio_3<da:00.0 NVMe Datacenter SSD [Optane]>

2. Collect metrics for all I/O devices below IIO stack

   ./perf stat --iiostat -- dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct
     357708+0 records in
     357707+0 records out
     375083606016 bytes (375 GB, 349 GiB) copied, 215.381 s, 1.7 GB/s

Performance counter stats for 'system wide':

device Inbound Read(MB) Inbound Write(MB) Outbound Read(MB) Outbound Write(MB)
00:00.0 0 0 0 0
81:00.0 0 0 0 0
18:00.0 0 0 0 0
86:00.0 0 0 0 0
88:00.0 0 0 0 0
3b:00.0 3 0 0 0
3c:03.0 3 0 0 0
3d:00.0 3 0 0 0
af:00.0 0 0 0 0
da:00.0 358559 44 0 22

215.383783574 seconds time elapsed

3. Collect metrics for comma separted list of I/O devices

   ./perf stat --iiostat=da:00.0 -- dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct
     381555+0 records in
     381554+0 records out
     400088457216 bytes (400 GB, 373 GiB) copied, 374.044 s, 1.1 GB/s

Performance counter stats for 'system wide':

device Inbound Read(MB) Inbound Write(MB) Outbound Read(MB) Outbound Write(MB)
da:00.0 382462 47 0 23

374.045775505 seconds time elapsed

Roman Sudarikov (2):
perf x86: Infrastructure for exposing an Uncore unit to PMON mapping perf x86: Exposing an Uncore unit to PMON for Intel Xeon® server
platform

  arch/x86/events/intel/uncore.c       |  39 ++++++-
  arch/x86/events/intel/uncore.h       |  10 +-
  arch/x86/events/intel/uncore_snbep.c | 162 +++++++++++++++++++++++++++
  3 files changed, 208 insertions(+), 3 deletions(-)

base-commit: 219d54332a09e8d8741c1e1982f5eae56099de85