Skip to contentSkip to conversation

PCI: s390 global attribute "UID Checking"

RFC Superseded Niklas Schnelle 4 participants 22 messages Jan 11, 2021 – Jan 21, 2021

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Niklas Schnelle

author maintainer
#0 lore raw

cover PCI: s390 global attribute "UID Checking"

Hi Bjorn, Hi Kernel Hackers,

With the below patch I'm proposing to expose a global (i.e. not device bound) runtime attribute of the s390 PCI implementation (zPCI) called "UID Checking". You can find some background information on what this attribute means and why it is important at the end of this mail. The reason I'm writing to you about it however is that this is the first global PCI attribute we would like to expose to user space and I'm searching for a good place to put it.
The proposed patch uses "/sys/bus/pci/zpci/uid_checking" which from our perspective would be a great choice but I realize that there currently are no platform specific attributes directly under "/sys/bus/pci" so this clearly requires some discussion. What's your thought on this and do you know of any other platform specific global PCI attributes as I couldn't find any?

Best regards,
Niklas Schnelle

Background:
On s390 OSs always run under at least a machine level hypervisor (LPAR). Simpliefied by usually running from SAN this makes VM migration possible at every level. For PCI this has created the need to allow PCI IDs to be stable across machines and to be partly user defined such that the setup of an existing VM on one machine can be recreated on another machine.
In particular the Domain part of the PCI ID can be user defined by setting a per device value called UID. Since this was a late addition and isn't used by all OSs and since UIDs need to be unique if used as Domains, there is an additional global platform supplied runtime value called "UID Checking". This value indicates if UIDs are guaranteed to be unique and set which triggers Linux to use the UIDs as PCI Domains otherwise PCI Domains are simply incremented as necessary. This "UID Checking" setting is thus very important e.g. when deciding how network interface names are generated as it indicates if the domain part of the PCI ID will remain stable across reboots and migrations. Once exposed we will propose a patch to udev/systemd to use the "UID Checking" attribute to prefer network interface names which can be guaranteed to be stable and re-creatable for migration.

Niklas Schnelle (1):
s390/pci: expose UID checking state in sysfs

 Documentation/ABI/testing/sysfs-bus-pci | 11 ++++++++
 arch/s390/include/asm/pci.h             |  3 +++
 arch/s390/pci/pci.c                     |  4 +++
 arch/s390/pci/pci_sysfs.c               | 34 +++++++++++++++++++++++++
 4 files changed, 52 insertions(+)
Signature · 1 line
2.17.1
author maintainer
#1 lore raw

RFC 1/1 s390/pci: expose UID checking state in sysfs

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

---
 Documentation/ABI/testing/sysfs-bus-pci | 11 ++++++++
 arch/s390/include/asm/pci.h             |  3 +++
 arch/s390/pci/pci.c                     |  4 +++
 arch/s390/pci/pci_sysfs.c               | 34 +++++++++++++++++++++++++
 4 files changed, 52 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +11 -0
index 25c9c39770c6..a174aac0ebb0 100644
@@ -375,3 +375,14 @@ Description:
 		The value comes from the PCI kernel device state and can be one
 		of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold".
 		The file is read only.
+What:		/sys/bus/pci/zpci/uid_checking
+Date:		December 2020
+Contact:	Niklas Schnelle <[email protected]>
+Description:
+		This attribute exposes the global state of UID Checking on
+		an s390 Linux system. If UID Checking is on this file
+		contains '1' otherwise '0'. If UID Checking is on the UID of
+		a zPCI device, or the UID of function zero for a multi-function
+		device will be used as its PCI Domain number. If UID Checking
+		is off PCI Domain numbers are generated automatically and
+		are not stable across reboots.
arch/s390/include/asm/pci.h +3 -0
index 212628932ddc..3cfa6cc701ba 100644
@@ -285,6 +285,9 @@ void zpci_debug_exit_device(struct zpci_dev *);
 /* Error reporting */
 int zpci_report_error(struct pci_dev *, struct zpci_report_error_header *);
 
+/* Sysfs Entries */
+int zpci_sysfs_init(void);
+
 #ifdef CONFIG_NUMA
 
 /* Returns the node based on PCI bus */
arch/s390/pci/pci.c +4 -0
index 41df8fcfddde..c16c93e5f9af 100644
@@ -881,6 +881,10 @@ static int __init pci_base_init(void)
 	if (rc)
 		goto out_find;
 
+	rc = zpci_sysfs_init();
+	if (rc)
+		goto out_find;
+
 	s390_pci_initialized = 1;
 	return 0;
 
arch/s390/pci/pci_sysfs.c +34 -0
index 5c028bee91b9..d00690f73539 100644
@@ -172,3 +172,37 @@ const struct attribute_group *zpci_attr_groups[] = {
 	&pfip_attr_group,
 	NULL,
 };
+
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);
+
+static struct kset *zpci_global_kset;
+
+static struct attribute *zpci_attrs_global[] = {
+	&sys_zpci_uid_checking_attr.attr,
+	NULL,
+};
+
+static struct attribute_group zpci_attr_group_global = {
+	.attrs = zpci_attrs_global,
+};
+
+int __init zpci_sysfs_init(void)
+{
+	struct kset *pci_bus_kset;
+
+	pci_bus_kset = bus_get_kset(&pci_bus_type);
+
+	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);
+	if (!zpci_global_kset)
+		return -ENOMEM;
+
+	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);
+}
Signature · 1 line
2.17.1
#2 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Cosmetic: can't tell if the above is two paragraphs separated by blank lines or four separated by either blank lines or short last lines. Please separate or reflow to avoid the ambiguity.

I don't have any input on the s390 issues, and I assume this will go via the s390 tree.

Quoted 64 lines
---
 Documentation/ABI/testing/sysfs-bus-pci | 11 ++++++++
 arch/s390/include/asm/pci.h             |  3 +++
 arch/s390/pci/pci.c                     |  4 +++
 arch/s390/pci/pci_sysfs.c               | 34 +++++++++++++++++++++++++
 4 files changed, 52 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +11 -0
index 25c9c39770c6..a174aac0ebb0 100644
@@ -375,3 +375,14 @@ Description:
 		The value comes from the PCI kernel device state and can be one
 		of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold".
 		The file is read only.
+What:		/sys/bus/pci/zpci/uid_checking
+Date:		December 2020
+Contact:	Niklas Schnelle <[email protected]>
+Description:
+		This attribute exposes the global state of UID Checking on
+		an s390 Linux system. If UID Checking is on this file
+		contains '1' otherwise '0'. If UID Checking is on the UID of
+		a zPCI device, or the UID of function zero for a multi-function
+		device will be used as its PCI Domain number. If UID Checking
+		is off PCI Domain numbers are generated automatically and
+		are not stable across reboots.
arch/s390/include/asm/pci.h +3 -0
index 212628932ddc..3cfa6cc701ba 100644
@@ -285,6 +285,9 @@ void zpci_debug_exit_device(struct zpci_dev *);
 /* Error reporting */
 int zpci_report_error(struct pci_dev *, struct zpci_report_error_header *);
 
+/* Sysfs Entries */
+int zpci_sysfs_init(void);
+
 #ifdef CONFIG_NUMA
 
 /* Returns the node based on PCI bus */
arch/s390/pci/pci.c +4 -0
index 41df8fcfddde..c16c93e5f9af 100644
@@ -881,6 +881,10 @@ static int __init pci_base_init(void)
 	if (rc)
 		goto out_find;
 
+	rc = zpci_sysfs_init();
+	if (rc)
+		goto out_find;
+
 	s390_pci_initialized = 1;
 	return 0;
 
arch/s390/pci/pci_sysfs.c +0 -0
index 5c028bee91b9..d00690f73539 100644
@@ -172,3 +172,37 @@ const struct attribute_group *zpci_attr_groups[] = {
 	&pfip_attr_group,
 	NULL,
 };

arch/s390/pci/pci_sysfs.c const struct attribute_group *zpci_attr_groups[] = { · line 184

Quoted 10 lines
arch/s390/pci/pci_sysfs.c +10 -0
+
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 26 lines
arch/s390/pci/pci_sysfs.c +23 -0
+static struct kset *zpci_global_kset;
+
+static struct attribute *zpci_attrs_global[] = {
+	&sys_zpci_uid_checking_attr.attr,
+	NULL,
+};
+
+static struct attribute_group zpci_attr_group_global = {
+	.attrs = zpci_attrs_global,
+};
+
+int __init zpci_sysfs_init(void)
+{
+	struct kset *pci_bus_kset;
+
+	pci_bus_kset = bus_get_kset(&pci_bus_type);
+
+	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);
+	if (!zpci_global_kset)
+		return -ENOMEM;
+
+	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);
+}
Signature · 1 line
2.17.1
author maintainer
#3 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 3 lines

Cosmetic: can't tell if the above is two paragraphs separated by blank lines or four separated by either blank lines or short last lines. Please separate or reflow to avoid the ambiguity.

Thanks, you're right I split it in 3 proper paragraphs now. Also the commit message was out of sync with the documentation, cover letter and code. This version actually uses /sys/bus/pci/zpci/uid_checking sorry about that.

Quoted 38 lines

I don't have any input on the s390 issues, and I assume this will go via the s390 tree.

Quoted 34 lines
---
 Documentation/ABI/testing/sysfs-bus-pci | 11 ++++++++
 arch/s390/include/asm/pci.h             |  3 +++
 arch/s390/pci/pci.c                     |  4 +++
 arch/s390/pci/pci_sysfs.c               | 34 +++++++++++++++++++++++++
 4 files changed, 52 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +11 -0
index 25c9c39770c6..a174aac0ebb0 100644
@@ -375,3 +375,14 @@ Description:
 		The value comes from the PCI kernel device state and can be one
 		of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold".
 		The file is read only.
+What:		/sys/bus/pci/zpci/uid_checking
+Date:		December 2020
+Contact:	Niklas Schnelle <[email protected]>
+Description:
+		This attribute exposes the global state of UID Checking on
+		an s390 Linux system. If UID Checking is on this file
+		contains '1' otherwise '0'. If UID Checking is on the UID of
+		a zPCI device, or the UID of function zero for a multi-function
+		device will be used as its PCI Domain number. If UID Checking
+		is off PCI Domain numbers are generated automatically and
+		are not stable across reboots.
arch/s390/include/asm/pci.h +0 -0
index 212628932ddc..3cfa6cc701ba 100644
@@ -285,6 +285,9 @@ void zpci_debug_exit_device(struct zpci_dev *);
 /* Error reporting */
 int zpci_report_error(struct pci_dev *, struct zpci_report_error_header *);

... snip ...

Quoted 4 lines
Quoted 4 lines
arch/s390/include/asm/pci.h +4 -0
+
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
Quoted 8 lines

arch/s390/include/asm/pci.h

Quoted 6 lines
arch/s390/include/asm/pci.h +6 -0
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Quoted 1 line

... snip ...

Quoted 1 line
Quoted 1 line
#4 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

arch/s390/pci/pci_sysfs.c const struct attribute_group *zpci_attr_groups[] = { · line 184

Quoted 9 lines
arch/s390/pci/pci_sysfs.c +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Bjorn

author maintainer
#5 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 36 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

arch/s390/pci/pci_sysfs.c const struct attribute_group *zpci_attr_groups[] = { · line 184

Quoted 9 lines
arch/s390/pci/pci_sysfs.c +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Quoted 6 lines

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch for that move should it be desired.

@Bjorn apart from the correct macro do you have a preference for either suggested path /sys/bus/pci/zpci/uid_checking vs /sys/bus/pci/uid_checking?

For completeness some further internal discussion lets us tend to rather name it to "unique_uids" but I guess that doesn't make a difference to non s390 people ;-)

#6 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 43 lines · Niklas Schnelle
Quoted 40 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

arch/s390/pci/pci_sysfs.c const struct attribute_group *zpci_attr_groups[] = { · line 184

Quoted 9 lines
arch/s390/pci/pci_sysfs.c +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Quoted 7 lines
Quoted 2 lines

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

Christian

#7 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 52 lines · Christian Brauner
Quoted 43 lines · Niklas Schnelle
Quoted 40 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

arch/s390/pci/pci_sysfs.c const struct attribute_group *zpci_attr_groups[] = { · line 184

Quoted 9 lines
arch/s390/pci/pci_sysfs.c +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Quoted 7 lines
Quoted 2 lines

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch

Quoted 7 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

author maintainer
#8 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 67 lines · Greg Kroah-Hartman
Quoted 52 lines · Christian Brauner
Quoted 43 lines · Niklas Schnelle
Quoted 40 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Quoted 7 lines
Quoted 2 lines

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch

Quoted 7 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

Quoted 7 lines

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

I'm aware that this is quite unusual in fact I couldn't find anything similar. That's why this is an RFC, with a lengthy cover letter explaining our use case, that I sent to Bjorn to figure out where to even place the attribute.

So I guess this is indeed me asking you how to do this properly. That said it does not show up under /sys/devices/* only /sys/bus/pci/*.

Best regards,
Niklas

#9 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 76 lines · Niklas Schnelle
Quoted 73 lines · Greg Kroah-Hartman
Quoted 52 lines · Christian Brauner
Quoted 43 lines · Niklas Schnelle
Quoted 40 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Quoted 7 lines
Quoted 2 lines

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch

Quoted 7 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

Quoted 7 lines
Quoted 1 line

greg k-h

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

Where _exactly_ are you wanting to put this attribute?

Quoted 7 lines

I'm aware that this is quite unusual in fact I couldn't find anything similar. That's why this is an RFC, with a lengthy cover letter explaining our use case, that I sent to Bjorn to figure out where to even place the attribute.

So I guess this is indeed me asking you how to do this properly. That said it does not show up under /sys/devices/* only /sys/bus/pci/*.

Do NOT put random kobjects under a bus subsystem. If you need that, then use BUS_ATTR_* as that is what it is there for.

Again, if you are in a driver subsystem, do not use a raw kobject. Either something is already there for you, or what you want to do is not correct :)

thanks,

greg k-h

author maintainer
#10 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 79 lines · Greg Kroah-Hartman
Quoted 78 lines · Niklas Schnelle
Quoted 74 lines · Greg Kroah-Hartman
Quoted 52 lines · Christian Brauner
Quoted 43 lines · Niklas Schnelle
Quoted 40 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Quoted 7 lines
Quoted 2 lines

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch

Quoted 7 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Quoted 7 lines
Quoted 5 lines

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

That is very possible.

Quoted 2 lines

Where _exactly_ are you wanting to put this attribute?

I'm trying for /sys/bus/pci/zpci/uid_checking, I'm using the below code and the attribute even shows up but reading it gives me two 0 bytes only.
The relevant code is only a slight alteration of the original patch as follows:

static ssize_t uid_checking_show(struct bus_type *bus, char *buf)

{
	return sprintf(buf, "%i\n", zpci_unique_uid);
}

static BUS_ATTR_RO(uid_checking);

static struct kset *zpci_global_kset;

static struct attribute *zpci_attrs_global[] = {
&bus_attr_uid_checking.attr,
NULL,
};

static struct attribute_group zpci_attr_group_global = {
.attrs = zpci_attrs_global,
};

int __init zpci_sysfs_init(void)

{
	struct kset *pci_bus_kset;

	pci_bus_kset = bus_get_kset(&pci_bus_type);

	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);
}
Quoted 9 lines
Quoted 8 lines

I'm aware that this is quite unusual in fact I couldn't find anything similar. That's why this is an RFC, with a lengthy cover letter explaining our use case, that I sent to Bjorn to figure out where to even place the attribute.

So I guess this is indeed me asking you how to do this properly. That said it does not show up under /sys/devices/* only /sys/bus/pci/*.

Quoted 6 lines

Do NOT put random kobjects under a bus subsystem. If you need that, then use BUS_ATTR_* as that is what it is there for.

Again, if you are in a driver subsystem, do not use a raw kobject. Either something is already there for you, or what you want to do is not correct :)

Understood and thanks for the clear advice!

Quoted 5 lines

thanks,

greg k-h

#11 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 109 lines · Niklas Schnelle
Quoted 79 lines · Greg Kroah-Hartman
Quoted 78 lines · Niklas Schnelle
Quoted 74 lines · Greg Kroah-Hartman
Quoted 52 lines · Christian Brauner
Quoted 43 lines · Niklas Schnelle
Quoted 40 lines · Bjorn Helgaas
Quoted 25 lines · Niklas Schnelle
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Niklas Schnelle

We use the UID of a zPCI adapter, or the UID of the function zero if there are multiple functions in an adapter, as PCI domain if and only if UID Checking is turned on.
Otherwise we automatically generate domains as devices appear.

The state of UID Checking is thus essential to know if the PCI domain will be stable, yet currently there is no way to access this information from userspace.
So let's solve this by showing the state of UID checking as a sysfs attribute in /sys/bus/pci/uid_checking

Quoted 11 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+/* Global zPCI attributes */
+static ssize_t uid_checking_show(struct kobject *kobj,
+				 struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%i\n", zpci_unique_uid);
+}
+
+static struct kobj_attribute sys_zpci_uid_checking_attr =
+	__ATTR(uid_checking, 0444, uid_checking_show, NULL);

Use DEVICE_ATTR_RO instead of __ATTR.

Quoted 9 lines

It's my understanding that DEVICE_ATTR_* is only for per device attributes. This one is global for the entire Z PCI. I just tried with BUS_ATTR_RO instead and that works but only if I put the attribute at /sys/bus/pci/uid_checking instead of with a zpci subfolder. This path would work for us too, we currently don't have any other global attributes that we are planning to expose but those could of course come up in the future.

Ah, I missed the fact that this is a kobj_attribute, not a device_attribute. Maybe KERNEL_ATTR_RO()? Very few uses so far, but seems like it might fit?

Quoted 7 lines
Quoted 2 lines

Bjorn

KERNEL_ATTR_* is currently not exported in any header. After adding it to include/linuc/sysfs.h it indeed works perfectly. Adding Christian Brauner as suggested by get_maintainers for their opinion. I'm of course willing to provide a patch

Quoted 7 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Quoted 7 lines
Quoted 5 lines

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

That is very possible.

Quoted 2 lines

Where _exactly_ are you wanting to put this attribute?

I'm trying for /sys/bus/pci/zpci/uid_checking, I'm using the below code and the attribute even shows up but reading it gives me two 0 bytes only.
The relevant code is only a slight alteration of the original patch as follows:

static ssize_t uid_checking_show(struct bus_type *bus, char *buf)

{
	return sprintf(buf, "%i\n", zpci_unique_uid);
}

static BUS_ATTR_RO(uid_checking);

static struct kset *zpci_global_kset;

Quoted 8 lines

static struct attribute *zpci_attrs_global[] = {
&bus_attr_uid_checking.attr,
NULL,
};

static struct attribute_group zpci_attr_group_global = {
.attrs = zpci_attrs_global,
};

Name your attribute group, and then you do not have to mess with a "raw" kobject like you are below:

Quoted 8 lines

int __init zpci_sysfs_init(void)

{
	struct kset *pci_bus_kset;

	pci_bus_kset = bus_get_kset(&pci_bus_type);

	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);

No, do not mess with at kset, just set the default attribute group for the bus to the above, and you should be fine.

Quoted 4 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

thanks,

greg k-h

author maintainer
#12 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Niklas Schnelle
Quoted 11 lines · Greg Kroah-Hartman
Quoted 10 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Christian Brauner
Quoted 5 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 55 lines
Quoted 55 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Quoted 7 lines
Quoted 5 lines

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

That is very possible.

Quoted 2 lines

Where _exactly_ are you wanting to put this attribute?

I'm trying for /sys/bus/pci/zpci/uid_checking, I'm using the below code and the attribute even shows up but reading it gives me two 0 bytes only.
The relevant code is only a slight alteration of the original patch as follows:

static ssize_t uid_checking_show(struct bus_type *bus, char *buf)

{
	return sprintf(buf, "%i\n", zpci_unique_uid);
}

static BUS_ATTR_RO(uid_checking);

static struct kset *zpci_global_kset;

static struct attribute *zpci_attrs_global[] = {
&bus_attr_uid_checking.attr,
NULL,
};

Quoted 6 lines
Quoted 3 lines

static struct attribute_group zpci_attr_group_global = {
.attrs = zpci_attrs_global,
};

Name your attribute group, and then you do not have to mess with a "raw" kobject like you are below:

Thanks for this tip and sorry for bothering you again.

Quoted 6 lines
Quoted 6 lines

int __init zpci_sysfs_init(void)

{
	struct kset *pci_bus_kset;
Quoted 6 lines
Quoted 3 lines
	pci_bus_kset = bus_get_kset(&pci_bus_type);

	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);

No, do not mess with at kset, just set the default attribute group for the bus to the above, and you should be fine.

Oh ok, I got this idea from the code adding /sys/bus/pci/slots/ in drivers/pci/slot.c:pci_slot_init(). See below maybe we can clean that up too.

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I think this would also allow us to get rid of pci_bus_get_kset() which is only used in that function and seems to me like it encourages use of raw ksets. Or I'm completely off the mark and just missing something important.

thanks,
Niklas

#13 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 91 lines · Niklas Schnelle
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Niklas Schnelle
Quoted 11 lines · Greg Kroah-Hartman
Quoted 10 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Christian Brauner
Quoted 5 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 55 lines
Quoted 55 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Quoted 7 lines
Quoted 5 lines

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

That is very possible.

Quoted 2 lines

Where _exactly_ are you wanting to put this attribute?

I'm trying for /sys/bus/pci/zpci/uid_checking, I'm using the below code and the attribute even shows up but reading it gives me two 0 bytes only.
The relevant code is only a slight alteration of the original patch as follows:

static ssize_t uid_checking_show(struct bus_type *bus, char *buf)

{
	return sprintf(buf, "%i\n", zpci_unique_uid);
}

static BUS_ATTR_RO(uid_checking);

static struct kset *zpci_global_kset;

static struct attribute *zpci_attrs_global[] = {
&bus_attr_uid_checking.attr,
NULL,
};

Quoted 6 lines
Quoted 3 lines

static struct attribute_group zpci_attr_group_global = {
.attrs = zpci_attrs_global,
};

Name your attribute group, and then you do not have to mess with a "raw" kobject like you are below:

Thanks for this tip and sorry for bothering you again.

Quoted 8 lines
Quoted 8 lines

int __init zpci_sysfs_init(void)

{
	struct kset *pci_bus_kset;

	pci_bus_kset = bus_get_kset(&pci_bus_type);
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);

No, do not mess with at kset, just set the default attribute group for the bus to the above, and you should be fine.

Oh ok, I got this idea from the code adding /sys/bus/pci/slots/ in drivers/pci/slot.c:pci_slot_init(). See below maybe we can clean that up too.

Slots are "interesting" and that code is really old, we know how to do things better now :)

But I doubt we should change anything there, as it does work, and userspace is used to how they come/go.

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Quoted 7 lines
Quoted 2 lines

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

Yes, add your group to that list of groups and all should be good.

Quoted 3 lines

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

No, use the above group you already have please.

Quoted 1 line

That said, I think we could also add something like bus_create_group().

No, use the group list as show above please.

Quoted 2 lines

Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

What needs to be cleaned up there?

Quoted 3 lines

I think this would also allow us to get rid of pci_bus_get_kset() which is only used in that function and seems to me like it encourages use of raw ksets. Or I'm completely off the mark and just missing something important.

Cleaning up slots is great, but they are "odd", so be careful.

thanks,

greg k-h

#14 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

In reply to Niklas Schnelle
Quoted 113 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 55 lines
Quoted 55 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Quoted 7 lines
Quoted 5 lines

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

That is very possible.

Quoted 2 lines

Where _exactly_ are you wanting to put this attribute?

I'm trying for /sys/bus/pci/zpci/uid_checking, I'm using the below code and the attribute even shows up but reading it gives me two 0 bytes only.
The relevant code is only a slight alteration of the original patch as follows:

static ssize_t uid_checking_show(struct bus_type *bus, char *buf)

{
	return sprintf(buf, "%i\n", zpci_unique_uid);
}

static BUS_ATTR_RO(uid_checking);

static struct kset *zpci_global_kset;

static struct attribute *zpci_attrs_global[] = {
&bus_attr_uid_checking.attr,
NULL,
};

Quoted 6 lines
Quoted 3 lines

static struct attribute_group zpci_attr_group_global = {
.attrs = zpci_attrs_global,
};

Name your attribute group, and then you do not have to mess with a "raw" kobject like you are below:

Thanks for this tip and sorry for bothering you again.

Quoted 6 lines
Quoted 6 lines

int __init zpci_sysfs_init(void)

{
	struct kset *pci_bus_kset;
Quoted 6 lines
Quoted 3 lines
	pci_bus_kset = bus_get_kset(&pci_bus_type);

	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);

No, do not mess with at kset, just set the default attribute group for the bus to the above, and you should be fine.

Oh ok, I got this idea from the code adding /sys/bus/pci/slots/ in drivers/pci/slot.c:pci_slot_init(). See below maybe we can clean that up too.

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

Bjorn

author maintainer
#15 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 119 lines · Bjorn Helgaas
Quoted 113 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 55 lines
Quoted 55 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines

Hey Niklas et al. :)

I think this will need input from Greg. He should be best versed in sysfs attributes. The problem with KERNEL_ATTR_* to me seems that it's supposed to be kernel internal. Now, that might just be a matter of renaming the macro but let's see whether Greg has any better idea or more questions. :)

The big question is, why are you needing this?

No driver or driver subsystem should EVER be messing with a "raw" kobject like this. Just use the existing DEVICE_* macros instead please.

If you are using a raw kobject, please ask me how to do this properly, as that is something that should NEVER show up in the /sys/devices/* tree. Otherwise userspace tools will break.

thanks,

greg k-h

Quoted 7 lines
Quoted 5 lines

Hi Greg,

this is for an architecture specific but global i.e. not device bound PCI attribute. That's why DEVICE_ATTR_* does not work. BUS_ATTR_* would work but only if we place the attribute directly under /sys/bus/pci/new_attr.

Then you are doing something wrong :)

That is very possible.

Quoted 2 lines

Where _exactly_ are you wanting to put this attribute?

I'm trying for /sys/bus/pci/zpci/uid_checking, I'm using the below code and the attribute even shows up but reading it gives me two 0 bytes only.
The relevant code is only a slight alteration of the original patch as follows:

static ssize_t uid_checking_show(struct bus_type *bus, char *buf)

{
	return sprintf(buf, "%i\n", zpci_unique_uid);
}

static BUS_ATTR_RO(uid_checking);

static struct kset *zpci_global_kset;

static struct attribute *zpci_attrs_global[] = {
&bus_attr_uid_checking.attr,
NULL,
};

Quoted 6 lines
Quoted 3 lines

static struct attribute_group zpci_attr_group_global = {
.attrs = zpci_attrs_global,
};

Name your attribute group, and then you do not have to mess with a "raw" kobject like you are below:

Thanks for this tip and sorry for bothering you again.

Quoted 6 lines
Quoted 6 lines

int __init zpci_sysfs_init(void)

{
	struct kset *pci_bus_kset;
Quoted 6 lines
Quoted 3 lines
	pci_bus_kset = bus_get_kset(&pci_bus_type);

	zpci_global_kset = kset_create_and_add("zpci", NULL, &pci_bus_kset->kobj);

No, do not mess with at kset, just set the default attribute group for the bus to the above, and you should be fine.

Oh ok, I got this idea from the code adding /sys/bus/pci/slots/ in drivers/pci/slot.c:pci_slot_init(). See below maybe we can clean that up too.

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

Quoted 5 lines

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

Yes you're right, in fact we internally also started to think about something more meaning oriented like "unique_uids". This indeed results in PCI addresses which can be relied upon as stable identifiers. For us it's enough that the domain matches as the bus is always 0 and dev/fn are determined by the device and SR-IOV stride etc. so will remain the same for equivalent configurations.

Quoted 10 lines

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

Bjorn

That's a very good point! I'm absolutely open to making this a common concept. I think the gist could be that the addressing/ids on the bus are reproducible, there might be some user configuration required but then they can be relied upon for stable names like network interfaces. So maybe a name like "reproducible_addressing"? I guess one non-s390 exclusive case of this could be virtualized PCI under QEMU/KVM etc. versus real hardware or even UEFI/Firmware guarantees, right?

author maintainer
#16 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

In reply to Bjorn Helgaas
Quoted 51 lines · Bjorn Helgaas
Quoted 34 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

Quoted 5 lines

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

Bjorn

Hi Bjorn,

I've thought about this more and even implemented a proof of concept patch for a global attribute using a pcibios_has_reproducible_addressing() hook.

However after implementing it I think as a more general and future proof concept it makes more sense to do this as a per device attribute, maybe as another flag in "stuct pci_dev" named something like "reliable_address". My reasoning behind this can be best be seen with a QEMU example. While I expect that QEMU can easily guarantee that one can always use "0000:01:00.0" for a virtio-pci NIC and thus enp1s0 interface name, the same might be harder to guarantee for a SR-IOV VF passed through with vfio-pci in that same VM and even less so if a thunderbolt controller is passed through and enumeration may depend on daisy chaining. The QEMU example also applies to s390 and maybe others will in the future. I'll send an RFC for a per device attribute but didn't want to let this discussion stand as if we had abandoned the idea and if you would prefer a global attribute I can also send an RFC for that.

Besst regards,
Niklas

#17 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

[Greg may be able to help compare/contrast this s390 UID with udev persistent names]

Quoted 59 lines · Niklas Schnelle
Quoted 46 lines · Bjorn Helgaas
Quoted 34 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

Quoted 7 lines

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

I've thought about this more and even implemented a proof of concept patch for a global attribute using a pcibios_has_reproducible_addressing() hook.

Quoted 11 lines

However after implementing it I think as a more general and future proof concept it makes more sense to do this as a per device attribute, maybe as another flag in "stuct pci_dev" named something like "reliable_address". My reasoning behind this can be best be seen with a QEMU example. While I expect that QEMU can easily guarantee that one can always use "0000:01:00.0" for a virtio-pci NIC and thus enp1s0 interface name, the same might be harder to guarantee for a SR-IOV VF passed through with vfio-pci in that same VM and even less so if a thunderbolt controller is passed through and enumeration may depend on daisy chaining. The QEMU example also applies to s390 and maybe others will in the future.

I'm a little wary of using the PCI geographical address ("0000:01:00.0") as a stable name. Even if you can make a way to use that to identify a specific device instance, regardless of how it is plugged in or passed through, it sounds like we could end up with "physical PCI addresses" and "virtual PCI addresses" that look the same and would cause confusion.

This concept sounds similar to the udev concept of a "persistent device name". What advantages does this s390 UID have over the udev approach?

There are optional PCI device serial numbers that we currently don't really make use of. Would that be a generic way to help with this?

#18 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 75 lines · Bjorn Helgaas

[Greg may be able to help compare/contrast this s390 UID with udev persistent names]

Quoted 59 lines · Niklas Schnelle
Quoted 46 lines · Bjorn Helgaas
Quoted 34 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

Quoted 7 lines

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

I've thought about this more and even implemented a proof of concept patch for a global attribute using a pcibios_has_reproducible_addressing() hook.

Quoted 11 lines

However after implementing it I think as a more general and future proof concept it makes more sense to do this as a per device attribute, maybe as another flag in "stuct pci_dev" named something like "reliable_address". My reasoning behind this can be best be seen with a QEMU example. While I expect that QEMU can easily guarantee that one can always use "0000:01:00.0" for a virtio-pci NIC and thus enp1s0 interface name, the same might be harder to guarantee for a SR-IOV VF passed through with vfio-pci in that same VM and even less so if a thunderbolt controller is passed through and enumeration may depend on daisy chaining. The QEMU example also applies to s390 and maybe others will in the future.

Quoted 6 lines

I'm a little wary of using the PCI geographical address ("0000:01:00.0") as a stable name. Even if you can make a way to use that to identify a specific device instance, regardless of how it is plugged in or passed through, it sounds like we could end up with "physical PCI addresses" and "virtual PCI addresses" that look the same and would cause confusion.

Agreed, as we all know, PCI addresses are never a stable name and can change every boot on some systems. Never rely on them, but you can use them as a "hint" for something that you have to determine is different from something else that is the same type of device.

Quoted 6 lines

This concept sounds similar to the udev concept of a "persistent device name". What advantages does this s390 UID have over the udev approach?

There are optional PCI device serial numbers that we currently don't really make use of. Would that be a generic way to help with this?

Only if you can require that they be unique. Is there such a requirement and who enforces it?

For USB, it was only "required" to have unique serial numbers for one class of devices (printers), and even then, it really wasn't enforced that much so you can't rely on it being unique at all, which makes it pretty useless :(

thanks,

greg k-h

author maintainer
#19 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

In reply to Bjorn Helgaas
Quoted 82 lines · Bjorn Helgaas

[Greg may be able to help compare/contrast this s390 UID with udev persistent names]

Quoted 59 lines · Niklas Schnelle
Quoted 46 lines · Bjorn Helgaas
Quoted 34 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line · Niklas Schnelle

On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

Quoted 7 lines

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

I've thought about this more and even implemented a proof of concept patch for a global attribute using a pcibios_has_reproducible_addressing() hook.

Quoted 11 lines

However after implementing it I think as a more general and future proof concept it makes more sense to do this as a per device attribute, maybe as another flag in "stuct pci_dev" named something like "reliable_address". My reasoning behind this can be best be seen with a QEMU example. While I expect that QEMU can easily guarantee that one can always use "0000:01:00.0" for a virtio-pci NIC and thus enp1s0 interface name, the same might be harder to guarantee for a SR-IOV VF passed through with vfio-pci in that same VM and even less so if a thunderbolt controller is passed through and enumeration may depend on daisy chaining. The QEMU example also applies to s390 and maybe others will in the future.

I'm a little wary of using the PCI geographical address ("0000:01:00.0") as a stable name. Even if you can make a way to use that to identify a specific device instance, regardless of how it is plugged in or passed through, it sounds like we could end up with "physical PCI addresses" and "virtual PCI addresses" that look the same and would cause confusion.

Quoted 7 lines

This concept sounds similar to the udev concept of a "persistent device name". What advantages does this s390 UID have over the udev approach?

There are optional PCI device serial numbers that we currently don't really make use of. Would that be a generic way to help with this?

As far as I understand systemd/udev uses the PCI geographical address by default ("enP<domain>p<bus>s<hotplug_slot_idx>...") for PCI attached network interfaces in many cases and a lot of users have already built their firewall/routing rules on these.

At the same time as you said this isn't ideal. On s390 things are a bit special since it's either really unstable, if UID Checking is not enforced (the value I wanted to expose is 0) the domain part and thus the interface name may change between reboots. On the other hand when it is enforced, the Domain can be defined in the machine configuration (IOCDS, think Domain XML but formatted for punch cards) and the bus is always 00.
The PCI geographical address and thus network interface names are then stable in the same way as with our CCW attached network interfaces where the CCW addresses have been stable for a long time and are regularly used for configuration. The interface would however still conform to the existing systemd/udev standard theme so the only change the user sees is that we would prefer the interface naming scheme which does not include the hotplug slot index (ehotplug slot indices are unique in the machine hypervisor and thus can't be the same in to guests on the same machine). We would thus not add a new name at all and thanks to the altname mechanism all existing rules including persistent naming via manual udev rules would keep working.

Now taking this beyond s390 my idea is that under some circumstances just as with UID Uniqueness for us, the platform can tell if a PCI geographical address is a reliable identifier thus sytemd/udev has more information about the quality of existing naming schemes incorporating information from the geographical address.

Looking at my personal KVM guests (Ubuntu, Arch Linux, Ubuntu ARM64) as well as my workstation (Arch Linux) all of them use a scheme with parts of the geographical address.

So in essence my idea is all about either choosing the best existing default name or making sure we at least know if it may not be reliable.

I hope that makes sense and I have added Lennart Poettering as he's the one everyone likes to blame for these names or at least can probably tell me what I missed.

Best,
Niklas

#20 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 89 lines · Niklas Schnelle
Quoted 86 lines · Bjorn Helgaas

[Greg may be able to help compare/contrast this s390 UID with udev persistent names]

Quoted 59 lines · Niklas Schnelle
Quoted 46 lines · Bjorn Helgaas
Quoted 34 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle
Quoted 2 lines · Bjorn Helgaas

On Wed, Jan 13, 2021 at 08:47:58AM +0100, Niklas Schnelle wrote:
On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

Quoted 7 lines

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

I've thought about this more and even implemented a proof of concept patch for a global attribute using a pcibios_has_reproducible_addressing() hook.

Quoted 11 lines

However after implementing it I think as a more general and future proof concept it makes more sense to do this as a per device attribute, maybe as another flag in "stuct pci_dev" named something like "reliable_address". My reasoning behind this can be best be seen with a QEMU example. While I expect that QEMU can easily guarantee that one can always use "0000:01:00.0" for a virtio-pci NIC and thus enp1s0 interface name, the same might be harder to guarantee for a SR-IOV VF passed through with vfio-pci in that same VM and even less so if a thunderbolt controller is passed through and enumeration may depend on daisy chaining. The QEMU example also applies to s390 and maybe others will in the future.

I'm a little wary of using the PCI geographical address ("0000:01:00.0") as a stable name. Even if you can make a way to use that to identify a specific device instance, regardless of how it is plugged in or passed through, it sounds like we could end up with "physical PCI addresses" and "virtual PCI addresses" that look the same and would cause confusion.

This concept sounds similar to the udev concept of a "persistent device name". What advantages does this s390 UID have over the udev approach?

Quoted 8 lines
Quoted 3 lines

There are optional PCI device serial numbers that we currently don't really make use of. Would that be a generic way to help with this?

As far as I understand systemd/udev uses the PCI geographical address by default ("enP<domain>p<bus>s<hotplug_slot_idx>...") for PCI attached network interfaces in many cases and a lot of users have already built their firewall/routing rules on these.

Which is fine as "normally" that does not change. But on some machines, it is quite volatile so users pick a different naming scheme.

And this is all done in userspace, I really don't understand what you want to do in the kernel here. If you want to expose another unique thing that the hardware knows about, wonderful, userspace can then use that if it wants to in how it names specific devices. But don't put that naming in the kernel, that's not where it belongs.

Quoted 5 lines

Now taking this beyond s390 my idea is that under some circumstances just as with UID Uniqueness for us, the platform can tell if a PCI geographical address is a reliable identifier thus sytemd/udev has more information about the quality of existing naming schemes incorporating information from the geographical address.

The platform does not "know" if this is reliable or not, sorry. That's not how PCI or UEFI works.

Quoted 3 lines

Looking at my personal KVM guests (Ubuntu, Arch Linux, Ubuntu ARM64) as well as my workstation (Arch Linux) all of them use a scheme with parts of the geographical address.

Because for the most part, yes, this works. Until you plug another device into the system. Or remove one. Or plug a hotplug device in and then cold boot with it plugged in (or removed). Or, my favorite system, just decide to renumber the PCI bus every other boot "just because".

None of that variability can be known by the kernel, that's only known by the user of that system, so again, they can make the best decision as to how to name their devices. If you want to use the systemd default, wonderful, but know that it does not work for everyone, so systemd allows you to do whatever you want.

Quoted 2 lines

So in essence my idea is all about either choosing the best existing default name or making sure we at least know if it may not be reliable.

There is no reliability "score" here, sorry. Hardware is fun :)

good luck!

greg k-h

author maintainer
#21 lore raw

Re: [RFC 1/1] s390/pci: expose UID checking state in sysfs

Quoted 99 lines · Greg Kroah-Hartman
Quoted 89 lines · Niklas Schnelle
Quoted 86 lines · Bjorn Helgaas

[Greg may be able to help compare/contrast this s390 UID with udev persistent names]

Quoted 59 lines · Niklas Schnelle
Quoted 46 lines · Bjorn Helgaas
Quoted 34 lines · Niklas Schnelle
Quoted 9 lines · Greg Kroah-Hartman
Quoted 8 lines · Niklas Schnelle
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Niklas Schnelle
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Christian Brauner
Quoted 3 lines · Niklas Schnelle

On 1/13/21 7:55 PM, Bjorn Helgaas wrote:
On Wed, Jan 13, 2021 at 08:47:58AM +0100, Niklas Schnelle wrote:
On 1/12/21 10:50 PM, Bjorn Helgaas wrote:

... snip ...

Quoted 11 lines
Quoted 5 lines
	if (!zpci_global_kset)
		return -ENOMEM;

	return sysfs_create_group(&zpci_global_kset->kobj, &zpci_attr_group_global);

Huge hint, if in a driver, or bus subsystem, and you call sysfs_*, that's usually a huge clue that you are doing something wrong.

Try the above again, with a simple attribute group, and name for it, and it should "just work".

I'm probably missing something but I don't get how this could work in this case. If I'm seeing this right the default attribute group here is pci_bus_type.bus_groups and that is already set in drivers/pci/pci-driver.c so I don't think I should set that.

I did however find bus_create_file() which does work when using the path /sys/bus/pci/uid_checking instead. This would work for us if Bjorn is okay with that path and the code is really clean and simple too.

Quoted 4 lines

That said, I think we could also add something like bus_create_group(). Then we could use that to also clean up drivers/pci/slot.c:pci_slot_init() and get the original path /sys/bus/pci/zpci/uid_checking.

I don't think "uid_checking" is quite the right name. It says something about the *implementation*, but it doesn't convey what that *means* to userspace. IIUC this file tells userspace something about whether a given PCI device always has the same PCI domain/bus/dev/fn address (or maybe just the same domain?)

Quoted 7 lines

It sounds like this feature could be useful beyond just s390, and other arches might implement it differently, without the UID concept. If so, I'm OK with something at the /sys/bus/pci/xxx level as long as the name is not s390-specific (and "uid" sounds s390-specific).

I assume it would also help with the udev/systemd end if you could make this less s390 dependent.

I've thought about this more and even implemented a proof of concept patch for a global attribute using a pcibios_has_reproducible_addressing() hook.

Quoted 11 lines

However after implementing it I think as a more general and future proof concept it makes more sense to do this as a per device attribute, maybe as another flag in "stuct pci_dev" named something like "reliable_address". My reasoning behind this can be best be seen with a QEMU example. While I expect that QEMU can easily guarantee that one can always use "0000:01:00.0" for a virtio-pci NIC and thus enp1s0 interface name, the same might be harder to guarantee for a SR-IOV VF passed through with vfio-pci in that same VM and even less so if a thunderbolt controller is passed through and enumeration may depend on daisy chaining. The QEMU example also applies to s390 and maybe others will in the future.

I'm a little wary of using the PCI geographical address ("0000:01:00.0") as a stable name. Even if you can make a way to use that to identify a specific device instance, regardless of how it is plugged in or passed through, it sounds like we could end up with "physical PCI addresses" and "virtual PCI addresses" that look the same and would cause confusion.

This concept sounds similar to the udev concept of a "persistent device name". What advantages does this s390 UID have over the udev approach?

Quoted 8 lines
Quoted 3 lines

There are optional PCI device serial numbers that we currently don't really make use of. Would that be a generic way to help with this?

As far as I understand systemd/udev uses the PCI geographical address by default ("enP<domain>p<bus>s<hotplug_slot_idx>...") for PCI attached network interfaces in many cases and a lot of users have already built their firewall/routing rules on these.

Quoted 8 lines

Which is fine as "normally" that does not change. But on some machines, it is quite volatile so users pick a different naming scheme.

And this is all done in userspace, I really don't understand what you want to do in the kernel here. If you want to expose another unique thing that the hardware knows about, wonderful, userspace can then use that if it wants to in how it names specific devices. But don't put that naming in the kernel, that's not where it belongs.

Oh no I definitely don't want to put any naming in the kernel. Rather this is very much "a thing that the hardware knows". The thing being:

We're a virtual platform and this PCI devices' address is generated from user configuration and we guarantee it's stable.

Quoted 9 lines
Quoted 6 lines

Now taking this beyond s390 my idea is that under some circumstances just as with UID Uniqueness for us, the platform can tell if a PCI geographical address is a reliable identifier thus sytemd/udev has more information about the quality of existing naming schemes incorporating information from the geographical address.

The platform does not "know" if this is reliable or not, sorry. That's not how PCI or UEFI works.

Yes I assumed as much which is why my examples were all about virtualized platforms/hypervisors. I would say QEMU very much knows if the PCI address is coming from its fluffy virtual sandbox or real wild hardware (pass-through).

Quoted 16 lines
Quoted 4 lines

Looking at my personal KVM guests (Ubuntu, Arch Linux, Ubuntu ARM64) as well as my workstation (Arch Linux) all of them use a scheme with parts of the geographical address.

Because for the most part, yes, this works. Until you plug another device into the system. Or remove one. Or plug a hotplug device in and then cold boot with it plugged in (or removed). Or, my favorite system, just decide to renumber the PCI bus every other boot "just because".

None of that variability can be known by the kernel, that's only known by the user of that system, so again, they can make the best decision as to how to name their devices. If you want to use the systemd default, wonderful, but know that it does not work for everyone, so systemd allows you to do whatever you want.

Quoted 4 lines
Quoted 2 lines

So in essence my idea is all about either choosing the best existing default name or making sure we at least know if it may not be reliable.

There is no reliability "score" here, sorry. Hardware is fun :)

Well as said above, at the very least there is real hardware, emulated hardware and our always virtualizing machines that keep things cozy and reliable enough that you can keep running OSs from the 70s largely unchanged on >5 GHz CPUs. So I think at least our platform has a pretty good track record in this regard.

Quoted 5 lines

good luck!

greg k-h