arm64: handle heterogeneous system register values

The features are printed per-cpu to match the format used by other architectures, and are derived from the (globally uniform) hwcaps. In cases where this may report incorrect information, rework is required elsewhere to function with varying instruction set support, and the sanity checks should provide us with some advance notice (warnings and TAINT_CPU_OUT_OF_SPEC). If we're lucky, such systems will never exist.

arch/arm64/kernel/setup.c +2 -2

-	for_each_online_cpu(i) {
 		/*
 		 * glibc reads /proc/cpuinfo to determine the number of
 		 * online processors, looking for lines beginning with
 		 * "processor".  Give glibc what it expects.
 		 */
 #ifdef CONFIG_SMP
-		seq_printf(m, "processor\t: %d\n", i);
+		seq_printf(m, "processor\t: %d\n", c);
 #endif
+		seq_printf(m, "implementer\t: 0x%02x\n",

Quoted 6 lines · Mark Rutland

The features are printed per-cpu to match the format used by other architectures, and are derived from the (globally uniform) hwcaps. In cases where this may report incorrect information, rework is required elsewhere to function with varying instruction set support, and the sanity checks should provide us with some advance notice (warnings and TAINT_CPU_OUT_OF_SPEC). If we're lucky, such systems will never exist.

[...]

Quoted 11 lines

arch/arm64/kernel/setup.c +2 -2

-	for_each_online_cpu(i) {
 		/*
 		 * glibc reads /proc/cpuinfo to determine the number of
 		 * online processors, looking for lines beginning with
 		 * "processor".  Give glibc what it expects.
 		 */
 #ifdef CONFIG_SMP
-		seq_printf(m, "processor\t: %d\n", i);
+		seq_printf(m, "processor\t: %d\n", c);
 #endif
+		seq_printf(m, "implementer\t: 0x%02x\n",

arch/arm64/kernel/setup.c static const char *hwcap_str[] = { · line 475

Quoted 10 lines

arch/arm64/kernel/setup.c +10 -0

+			   MIDR_IMPLEMENTOR(midr));
+		seq_printf(m, "variant\t\t: 0x%x\n", MIDR_VARIANT(midr));
+		seq_printf(m, "partnum\t\t: 0x%03x\n", MIDR_PARTNUM(midr));
+		seq_printf(m, "revision\t: 0x%x\n", MIDR_REVISION(midr));
+
+		/* dump out the processor features */
+		seq_puts(m, "features\t: ");
+		for (i = 0; hwcap_str[i]; i++)
+			if (elf_hwcap & (1 << i))
+				seq_printf(m, "%s ", hwcap_str[i]);

I don't have hugely strong opinions about this, but I don't see why it's useful to print exactly the same line out `n' times; once for each CPU. We only pass one set of hwcaps to ELF executables via auxv, so why do we need to duplicate things here?

Put another way, we're really treating the hwcaps as a system property rather than a per-cpu property, so I think we should handle them as such.

I don't have hugely strong opinions about this, but I don't see why it's useful to print exactly the same line out `n' times; once for each CPU. We only pass one set of hwcaps to ELF executables via auxv, so why do we need to duplicate things here?

Put another way, we're really treating the hwcaps as a system property rather than a per-cpu property, so I think we should handle them as such.

Quoted 4 lines · Will Deacon

I don't have hugely strong opinions about this, but I don't see why it's useful to print exactly the same line out `n' times; once for each CPU. We only pass one set of hwcaps to ELF executables via auxv, so why do we need to duplicate things here?

I think there are a couple of lines of argument here:

(1) Precedent from other architectures. Both 32-bit ARM and x86 have the feature-info be per-core; we should do the same for 64-bit ARM unless there's a really strong reason not to.

(2) Making this be not-per-CPU backs us into a corner. If we have the feature flags per-core, and it turns out that they're never ever different between cores even on heterogenous CPUs, there's no problem. If we have the feature flags be listed only once, and then in future we do want to support some minor form of heterogeny between cores, then we're

stuck with a compatibility break because the format doesn't let us express that. (Perhaps there might be a big.LITTLE system where only the big cores had the crypto extensions, to pick a random possibility where the kernel doesn't need to care but userspace could take advantage by restricting itself to running on only the big cores). I think this makes the choice pretty straightforward, since there's basically no benefit to having only a single features line.

Quoted 6 lines · Peter Maydell

Quoted 4 lines · Will Deacon

I don't have hugely strong opinions about this, but I don't see why it's useful to print exactly the same line out `n' times; once for each CPU. We only pass one set of hwcaps to ELF executables via auxv, so why do we need to duplicate things here?

Quoted 5 lines

I think there are a couple of lines of argument here:

(1) Precedent from other architectures. Both 32-bit ARM and x86 have the feature-info be per-core; we should do the same for 64-bit ARM unless there's a really strong reason not to.

Agreed.

Quoted 6 lines

(2) Making this be not-per-CPU backs us into a corner. If we have the feature flags per-core, and it turns out that they're never ever different between cores even on heterogenous CPUs, there's no problem. If we have the feature flags be listed only once, and then in future we do want to support some minor form of heterogeny between cores, then we're

Quoted 8 lines

stuck with a compatibility break because the format doesn't let us express that. (Perhaps there might be a big.LITTLE system where only the big cores had the crypto extensions, to pick a random possibility where the kernel doesn't need to care but userspace could take advantage by restricting itself to running on only the big cores). I think this makes the choice pretty straightforward, since there's basically no benefit to having only a single features line.

Agreed.

Quoted 9 lines · Marcus Shawcroft

Quoted 7 lines · Peter Maydell

Quoted 5 lines · Will Deacon

I don't have hugely strong opinions about this, but I don't see why it's useful to print exactly the same line out `n' times; once for each CPU. We only pass one set of hwcaps to ELF executables via auxv, so why do we need to duplicate things here?

Quoted 7 lines

Quoted 5 lines

I think there are a couple of lines of argument here:

(1) Precedent from other architectures. Both 32-bit ARM and x86 have the feature-info be per-core; we should do the same for 64-bit ARM unless there's a really strong reason not to.

Agreed.

I don't really see the benefits of pretending that /proc/cpuinfo is remotely portable between architectures.

Quoted 15 lines

Quoted 6 lines

(2) Making this be not-per-CPU backs us into a corner. If we have the feature flags per-core, and it turns out that they're never ever different between cores even on heterogenous CPUs, there's no problem. If we have the feature flags be listed only once, and then in future we do want to support some minor form of heterogeny between cores, then we're

Quoted 8 lines

stuck with a compatibility break because the format doesn't let us express that. (Perhaps there might be a big.LITTLE system where only the big cores had the crypto extensions, to pick a random possibility where the kernel doesn't need to care but userspace could take advantage by restricting itself to running on only the big cores). I think this makes the choice pretty straightforward, since there's basically no benefit to having only a single features line.

Quoted 1 line

Agreed.

So the real question is: do we want to allow Linux to support features that exist only on a subset of cores in the system? The current thinking is that we truncate the advertised features to the common system subset, which means it will be the same on all cores, by definition. That allows hardware guys to build crazy systems that we can at least use, without imparting a world of pain onto software that needs to run on them.

I also think that we're backed into a corner regardless. We've seen how people ignore hwcaps (e.g. SWP), so they're likely to parse the caps for CPU0 and use that as an indication of the system capabilities.

Note that the features list *isn't* just the features supported by the CPU. It also takes into account the set of features supported by the kernel (e.g. we don't advertise VFP on ARMv7 if VFP context switching isn't enabled).

I don't really see the benefits of pretending that /proc/cpuinfo is remotely portable between architectures.

So the real question is: do we want to allow Linux to support features that exist only on a subset of cores in the system? The current thinking is that we truncate the advertised features to the common system subset, which means it will be the same on all cores, by definition. That allows hardware guys to build crazy systems that we can at least use, without imparting a world of pain onto software that needs to run on them.

I also think that we're backed into a corner regardless. We've seen how people ignore hwcaps (e.g. SWP), so they're likely to parse the caps for CPU0 and use that as an indication of the system capabilities.

Note that the features list *isn't* just the features supported by the CPU. It also takes into account the set of features supported by the kernel (e.g. we don't advertise VFP on ARMv7 if VFP context switching isn't enabled).

Quoted 2 lines · Will Deacon

I don't really see the benefits of pretending that /proc/cpuinfo is remotely portable between architectures.

I'm not suggesting it's portable. I'm suggesting that you need a good reason to push backwards against a design decision made on other architectures. In particular, given that ARM is in general *more* likely to be heterogenous than x86 (given the existence of big.LITTLE), it seems baffling to try to move in a direction that denies the possibility of further heterogeneity in future.

Quoted 6 lines

So the real question is: do we want to allow Linux to support features that exist only on a subset of cores in the system? The current thinking is that we truncate the advertised features to the common system subset, which means it will be the same on all cores, by definition. That allows hardware guys to build crazy systems that we can at least use, without imparting a world of pain onto software that needs to run on them.

I've already made one suggestion (non-pervasive crypto). You could also envisage "feature bits" that effectively mean "things will be faster on this core" even if they still work on other cores too.

Quoted 3 lines

I also think that we're backed into a corner regardless. We've seen how people ignore hwcaps (e.g. SWP), so they're likely to parse the caps for CPU0 and use that as an indication of the system capabilities.

Certainly userspace *can* do the wrong thing. That doesn't necessarily mean that the kernel should only ever provide the API equivalent of safety scissors...

Quoted 3 lines

Note that the features list *isn't* just the features supported by the CPU. It also takes into account the set of features supported by the kernel (e.g. we don't advertise VFP on ARMv7 if VFP context switching isn't enabled).

This is an argument for also advertising the lowest-common-denominator feature bits. (Do you want to argue for a single "features:" line plus per-core "extra-features:" lines?)

We're not denying the possibility of heterogeneity, we're trying to expose a consistent view of the system to userspace. Differences between cores should be dealt with by the kernel (e.g. IKS, HMP scheduling), not blindly passed off to userspace.

Quoted 4 lines · Will Deacon

We're not denying the possibility of heterogeneity, we're trying to expose a consistent view of the system to userspace. Differences between cores should be dealt with by the kernel (e.g. IKS, HMP scheduling), not blindly passed off to userspace.

On that basis, why report anything at all about invididual cores? Just have /proc/cpuinfo report "number of processors: 4" and no per-CPU information at all...

Quoted 9 lines · Peter Maydell

Quoted 4 lines · Will Deacon

We're not denying the possibility of heterogeneity, we're trying to expose a consistent view of the system to userspace. Differences between cores should be dealt with by the kernel (e.g. IKS, HMP scheduling), not blindly passed off to userspace.

On that basis, why report anything at all about invididual cores? Just have /proc/cpuinfo report "number of processors: 4" and no per-CPU information at all...

We lost a lot of time on this already (given the internal threads). So my proposal is to go ahead with Mark's patch with per-CPU features. They currently just include the same elf_hwcap multiple times. If we ever need to present different features, the conditions would be:

1. Never report more than elf_hwcap
2. elf_hwcap can only include non-symmetric features *if* Linux gets a way to transparently handle migration or emulation

Quoted 16 lines · Catalin Marinas

Quoted 9 lines · Peter Maydell

Quoted 4 lines · Will Deacon

We're not denying the possibility of heterogeneity, we're trying to expose a consistent view of the system to userspace. Differences between cores should be dealt with by the kernel (e.g. IKS, HMP scheduling), not blindly passed off to userspace.

On that basis, why report anything at all about invididual cores? Just have /proc/cpuinfo report "number of processors: 4" and no per-CPU information at all...

We lost a lot of time on this already (given the internal threads). So my proposal is to go ahead with Mark's patch with per-CPU features. They currently just include the same elf_hwcap multiple times. If we ever need to present different features, the conditions would be:

Quoted 3 lines

1. Never report more than elf_hwcap
2. elf_hwcap can only include non-symmetric features *if* Linux gets a way to transparently handle migration or emulation

... making the point of a per-cpu field entirely pointless ;)

Quoted 17 lines · Will Deacon

Quoted 16 lines · Catalin Marinas

Quoted 9 lines · Peter Maydell

Quoted 4 lines · Will Deacon

We're not denying the possibility of heterogeneity, we're trying to expose a consistent view of the system to userspace. Differences between cores should be dealt with by the kernel (e.g. IKS, HMP scheduling), not blindly passed off to userspace.

On that basis, why report anything at all about invididual cores? Just have /proc/cpuinfo report "number of processors: 4" and no per-CPU information at all...

We lost a lot of time on this already (given the internal threads). So my proposal is to go ahead with Mark's patch with per-CPU features. They currently just include the same elf_hwcap multiple times. If we ever need to present different features, the conditions would be:

Quoted 5 lines

Quoted 3 lines

1. Never report more than elf_hwcap
2. elf_hwcap can only include non-symmetric features *if* Linux gets a way to transparently handle migration or emulation

... making the point of a per-cpu field entirely pointless ;)

Well, if we can support such features in a transparent way, /proc/cpuinfo becomes more informative (e.g. user wondering why a process runs only on certain CPUs).

But to be clear (and I think we are aligned), I don't trust user space to parse all processors in /proc/cpuinfo and make an informed selection of CPU affinity to avoid SIGILL.

Yet another option would be to have a single features/hwcap line and present the extra features in a human (and only human) readable form (e.g. some haiku that changes with every kernel release ;)).

Quoted 32 lines · Catalin Marinas

Quoted 17 lines · Will Deacon

Quoted 16 lines · Catalin Marinas

Quoted 9 lines · Peter Maydell

Quoted 4 lines · Will Deacon

We're not denying the possibility of heterogeneity, we're trying to expose a consistent view of the system to userspace. Differences between cores should be dealt with by the kernel (e.g. IKS, HMP scheduling), not blindly passed off to userspace.

On that basis, why report anything at all about invididual cores? Just have /proc/cpuinfo report "number of processors: 4" and no per-CPU information at all...

We lost a lot of time on this already (given the internal threads). So my proposal is to go ahead with Mark's patch with per-CPU features. They currently just include the same elf_hwcap multiple times. If we ever need to present different features, the conditions would be:

Quoted 5 lines

Quoted 3 lines

1. Never report more than elf_hwcap
2. elf_hwcap can only include non-symmetric features *if* Linux gets a way to transparently handle migration or emulation

... making the point of a per-cpu field entirely pointless ;)

Well, if we can support such features in a transparent way, /proc/cpuinfo becomes more informative (e.g. user wondering why a process runs only on certain CPUs).

But to be clear (and I think we are aligned), I don't trust user space to parse all processors in /proc/cpuinfo and make an informed selection of CPU affinity to avoid SIGILL.

Quoted 3 lines

Yet another option would be to have a single features/hwcap line and present the extra features in a human (and only human) readable form (e.g. some haiku that changes with every kernel release ;)).

Or just have the single features line, then the per-cpu line can be called `flags' or something, like Ard suggested. If userspace decides to parse flags, it deserves all the pain that it gets.

From 7dc7b5e5c4a54f56a5a7e59d2dd0de009dc9b9d0 Mon Sep 17 00:00:00 2001

Comments are added to guide userspace developers in the right direction (using the hwcaps provided in auxval). Hopefully where userspace applications parse /proc/cpuinfo rather than using the readily available hwcaps, they limit themselves to reading said first line.

Quoted 4 lines · Mark Rutland

Comments are added to guide userspace developers in the right direction (using the hwcaps provided in auxval). Hopefully where userspace applications parse /proc/cpuinfo rather than using the readily available hwcaps, they limit themselves to reading said first line.

Comments in the kernel sources aren't going to guide anybody except kernel developers.

I was expecting from this commit message that you were going to emit actual comments in /proc/cpuinfo...

Quoted 2 lines · Peter Maydell

Comments in the kernel sources aren't going to guide anybody except kernel developers.

That's not entirely true, some people skim the kernel sources to figure out how they're meant to use syscalls and such (though admitedly this isn't all that common).

Quoted 2 lines

I was expecting from this commit message that you were going to emit actual comments in /proc/cpuinfo...

I don't think that's a good idea, and I can only see that reading when I squint quite hard. ;)

Quoted 7 lines · Mark Rutland

Quoted 2 lines · Peter Maydell

Comments in the kernel sources aren't going to guide anybody except kernel developers.

That's not entirely true, some people skim the kernel sources to figure out how they're meant to use syscalls and such (though admitedly this isn't all that common).

Every time anybody has to do that it means you've failed to document something...

Quoted 5 lines

Quoted 2 lines

I was expecting from this commit message that you were going to emit actual comments in /proc/cpuinfo...

I don't think that's a good idea, and I can only see that reading when I squint quite hard. ;)

You have to admit it would put the documentation right where the people looking at cpuinfo can find it :-)

How about a patch to Documentation/filesystems/proc.txt ?

Quoted 11 lines · Peter Maydell

Quoted 7 lines · Mark Rutland

Quoted 2 lines · Peter Maydell

Comments in the kernel sources aren't going to guide anybody except kernel developers.

That's not entirely true, some people skim the kernel sources to figure out how they're meant to use syscalls and such (though admitedly this isn't all that common).

Every time anybody has to do that it means you've failed to document something...

...and in this case, the thing to document is the hwcaps.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

I was expecting from this commit message that you were going to emit actual comments in /proc/cpuinfo...

I don't think that's a good idea, and I can only see that reading when I squint quite hard. ;)

You have to admit it would put the documentation right where the people looking at cpuinfo can find it :-)

Sure :)

Quoted 1 line

How about a patch to Documentation/filesystems/proc.txt ?

Currently there seems to be a single relevant line, and it doesn't seem to be up-to-date for SMP:

cpuinfo Info about the CPU

It might make sense to have something under Documentation/arm64, but I don't know what precisely.

Currently the arm64 kernel assumes CPUs are homogeneous, which is a source of several potential problems as this is not always true. This series is an attempt to ameliorate the situation. The first posting [1] was titled as "arm64: extend/fix /proc/cpuinfo + runtime sanity checks", but with the addition of the I-cache policy fix this title was either going to become unwieldy or misleading.

Mark Rutland (5):
arm64: add MIDR_EL1 field accessors
arm64: cpuinfo: record cpu system register values
arm64: cachetype: report weakest cache policy
arm64: add runtime system sanity checks

arm64: cpuinfo: print info for all CPUs

Quoted 5 lines · Mark Rutland

Mark Rutland (5):
arm64: add MIDR_EL1 field accessors
arm64: cpuinfo: record cpu system register values
arm64: cachetype: report weakest cache policy
arm64: add runtime system sanity checks

For these 4 patches:

Reviewed-by Catalin Marinas <[email protected]>

Quoted 2 lines

arm64: cpuinfo: print info for all CPUs

This is still up for discussion.

Quoted 9 lines · Catalin Marinas

Quoted 5 lines · Mark Rutland

Mark Rutland (5):
arm64: add MIDR_EL1 field accessors
arm64: cpuinfo: record cpu system register values
arm64: cachetype: report weakest cache policy
arm64: add runtime system sanity checks

For these 4 patches:

Quoted 1 line

Reviewed-by Catalin Marinas <[email protected]>

Thanks.

I take it you'll pick these up directly and apply the tags from you and Will?

Or would you prefer I place these in a branch?