Skip to contentSkip to conversation

RISC-V: Detect Ssqosid extension and handle sqoscfg CSR

RFC PATCH In review Drew Fustini 4 participants 8 messages Apr 10, 2023 – Apr 21, 2023

Subsystems (2)
lore
Share & tools
lore
#1 lore raw

RFC PATCH 1/2 RISC-V: Detect the Ssqosid extension

In reply to Drew Fustini

From: Kornel Dulęba <[email protected]>

Detect the Ssqosid extension (Supervisor-mode Quality of Service ID) as defined in the CBQRI (Capacity and Bandwidth QoS Register Interface) specification.

---
Note: the Ssqosid extension and CBQRI spec are still in a draft state.

 arch/riscv/include/asm/hwcap.h | 2 ++
 arch/riscv/kernel/cpu.c        | 1 +
 arch/riscv/kernel/cpufeature.c | 1 +
 3 files changed, 4 insertions(+)
arch/riscv/include/asm/hwcap.h +2 -0
index 9af793970855..6dd717030723 100644
@@ -44,6 +44,8 @@
 #define RISCV_ISA_EXT_ZIHINTPAUSE	32
 #define RISCV_ISA_EXT_SVNAPOT		33
 #define RISCV_ISA_EXT_ZICBOZ		34
+#define RISCV_ISA_EXT_SSQOSID		35
+
 
 #define RISCV_ISA_EXT_MAX		64
 #define RISCV_ISA_EXT_NAME_LEN_MAX	32
arch/riscv/kernel/cpu.c +1 -0
index 9203e18320f9..04bd0a21910e 100644
@@ -190,6 +190,7 @@ static struct riscv_isa_ext_data isa_ext_arr[] = {
 	__RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE),
 	__RISCV_ISA_EXT_DATA(zbb, RISCV_ISA_EXT_ZBB),
 	__RISCV_ISA_EXT_DATA(sscofpmf, RISCV_ISA_EXT_SSCOFPMF),
+	__RISCV_ISA_EXT_DATA(ssqosid, RISCV_ISA_EXT_SSQOSID),
 	__RISCV_ISA_EXT_DATA(sstc, RISCV_ISA_EXT_SSTC),
 	__RISCV_ISA_EXT_DATA(svinval, RISCV_ISA_EXT_SVINVAL),
 	__RISCV_ISA_EXT_DATA(svnapot, RISCV_ISA_EXT_SVNAPOT),
arch/riscv/kernel/cpufeature.c +1 -0
index 00d7cd2c9043..94bf5556347d 100644
@@ -226,6 +226,7 @@ void __init riscv_fill_hwcap(void)
 			} else {
 				/* sorted alphabetically */
 				SET_ISA_EXT_MAP("sscofpmf", RISCV_ISA_EXT_SSCOFPMF);
+				SET_ISA_EXT_MAP("ssqosid", RISCV_ISA_EXT_SSQOSID);
 				SET_ISA_EXT_MAP("sstc", RISCV_ISA_EXT_SSTC);
 				SET_ISA_EXT_MAP("svinval", RISCV_ISA_EXT_SVINVAL);
 				SET_ISA_EXT_MAP("svnapot", RISCV_ISA_EXT_SVNAPOT);
Signature · 1 line
2.34.1
#0 lore raw

cover RISC-V: Detect Ssqosid extension and handle sqoscfg CSR

This RFC series adds initial support for the Ssqosid extension and the sqoscfg CSR as specified in Chapter 2 of the RISC-V Capacity and Bandwidth Controller QoS Register Interface (CBQRI) specification [1].

QoS (Quality of Service) in this context is concerned with shared resources on an SoC such as cache capacity and memory bandwidth. Intel and AMD already have QoS features on x86, and there is an existing user interface in Linux: the resctrl virtual filesystem [2].

The sqoscfg CSR provides a mechanism by which a software workload (e.g. a process or a set of processes) can be associated with a resource control ID (RCID) and a monitoring counter ID (MCID) that accompanies each request made by the hart to shared resources like cache. CBQRI defines operations to configure resource usage limits, in the form of capacity or bandwidth, for an RCID. CBQRI also defines operations to configure counters to track the resource utilization of an MCID.

The CBQRI spec is still in draft state and is undergoing review [3]. It is possible there will be changes to the Ssqosid extension and the CBQRI spec. For example, the CSR address for sqoscfg is not yet finalized.

My goal for this RFC is to determine if the 2nd patch is an acceptable approach to handling sqoscfg when switching tasks. This RFC was tested against a QEMU branch that implements the Ssqosid extension [4]. A test driver [5] was used to set thread_struct.sqoscfg for the current process. This allowed __qos_sched_in() to be tested without resctrl.

A forthcoming RFC series will enable resctrl to support systems that implement CBQRI and will reference a QEMU branch that implements CBQRI.

NOTE: CBQRI was previously known as CMQRI and the github repo with the spec has not yet been renamed [6].

[1] https://github.com/riscv-non-isa/riscv-cmqri/blob/main/riscv-cbqri.pdf
[2] https://docs.kernel.org/x86/resctrl.html
[3] https://lists.riscv.org/g/tech-cmqri/message/38
[4] https://github.com/pdp7/qemu/tree/ssqosid_v8.0.0-rc1
[5] https://github.com/pdp7/linux/commits/ssqosid_sqoscfg_v6.3
[6] https://lists.riscv.org/g/tech-cmqri/message/41

Drew Fustini (1):
RISC-V: Add support for sqoscfg CSR

Kornel Dulęba (1):
RISC-V: Detect the Ssqosid extension

 arch/riscv/Kconfig                 | 19 ++++++++++++++
 arch/riscv/include/asm/csr.h       |  8 ++++++
 arch/riscv/include/asm/hwcap.h     |  2 ++
 arch/riscv/include/asm/processor.h |  3 +++
 arch/riscv/include/asm/qos.h       | 40 ++++++++++++++++++++++++++++++
 arch/riscv/include/asm/switch_to.h |  2 ++
 arch/riscv/kernel/cpu.c            |  1 +
 arch/riscv/kernel/cpufeature.c     |  1 +
 8 files changed, 76 insertions(+)
 create mode 100644 arch/riscv/include/asm/qos.h

base-commit: d34a6b715a23ccd9c9d0bc7a475bea59dc3e28b2

Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH 2/2 RISC-V: Add support for sqoscfg CSR

Add support for the sqoscfg CSR defined in the Ssqosid ISA extension (Supervisor-mode Quality of Service ID). The CSR contains two fields:

- Resource Control ID (RCID) used determine resource allocation
- Monitoring Counter ID (MCID) used to track resource usage

Requests from a hart to shared resources like cache will be tagged with these IDs. This allows the usage of shared resources to be associated with the task currently running on the hart.

A sqoscfg field is added to thread_struct and has the same format as the sqoscfg CSR. This allows the scheduler to set the hart's sqoscfg CSR to contain the RCID and MCID for the task that is being scheduled in. The sqoscfg CSR is only written to if the thread_struct.sqoscfg is different from the current value of the CSR.

A per-cpu variable cpu_sqoscfg is used to mirror that state of the CSR. This is because access to L1D hot memory should be several times faster than a CSR read. Also, in the case of virtualization, accesses to this CSR are trapped in the hypervisor.

---
Note: the Ssqosid extension and CBQRI spec are still in a draft state. The CSR address of sqoscfg is not final.

Changes from the original patch [1]:
- Rebase from 6.0 to 6.3
- Simplify per-cpu variable from struct to u32 with just sqoscfg
- Move qoscfg to thread_struct in arch/riscv/include/asm/processor.h This avoids changing task_struct in /include/linux/sched.h
- Reword commit description
- Reword Kconfig description

[1] https://github.com/rivosinc/linux/commit/8454b793a62be21d39e5826ef5241dfa06198ba9

 arch/riscv/Kconfig                 | 19 ++++++++++++++
 arch/riscv/include/asm/csr.h       |  8 ++++++
 arch/riscv/include/asm/processor.h |  3 +++
 arch/riscv/include/asm/qos.h       | 40 ++++++++++++++++++++++++++++++
 arch/riscv/include/asm/switch_to.h |  2 ++
 5 files changed, 72 insertions(+)
 create mode 100644 arch/riscv/include/asm/qos.h
arch/riscv/Kconfig +19 -0
index cc02eb9eee1f..03f22b7fe34b 100644
@@ -418,6 +418,25 @@ config RISCV_ISA_SVNAPOT
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_SSQOSID
+	bool "Ssqosid extension support"
+	default y
+	help
+	  Adds support for the Ssqosid ISA extension (Supervisor-mode
+	  Quality of Service ID).
+
+	  Ssqosid defines the sqoscfg CSR which allows the system to tag
+	  the running process with RCID (Resource Control ID) and MCID
+	  (Monitoring Counter ID). The RCID is used determine resource
+	  allocation. The MCID is used to track resource usage in event
+	  counters.
+
+	  For example, a cache controller may use the RCID to apply a
+	  cache partitioning scheme and use the MCID to track how much
+	  cache a process, or a group of processes, is using.
+
+	  If you don't know what to do here, say Y.
+
 config RISCV_ISA_SVPBMT
 	bool "SVPBMT extension support"
 	depends on 64BIT && MMU
arch/riscv/include/asm/csr.h +8 -0
index 7c2b8cdb7b77..17d04a0cacd6 100644
@@ -59,6 +59,13 @@
 #define SATP_ASID_MASK	_AC(0xFFFF, UL)
 #endif
 
+/* SQOSCFG fields */
+#define SQOSCFG_RCID_MASK	_AC(0x00000FFF, UL)
+#define SQOSCFG_MCID_MASK	SQOSCFG_RCID_MASK
+#define SQOSCFG_MCID_SHIFT	16
+#define SQOSCFG_MASK		((SQOSCFG_MCID_MASK << SQOSCFG_MCID_SHIFT) | \
+				  SQOSCFG_RCID_MASK)
+
 /* Exception cause high bit - is an interrupt if set */
 #define CAUSE_IRQ_FLAG		(_AC(1, UL) << (__riscv_xlen - 1))
 
@@ -245,6 +252,7 @@
 #define CSR_STVAL		0x143
 #define CSR_SIP			0x144
 #define CSR_SATP		0x180
+#define CSR_SQOSCFG		0x181
 
 #define CSR_STIMECMP		0x14D
 #define CSR_STIMECMPH		0x15D
arch/riscv/include/asm/processor.h +3 -0
index 94a0590c6971..724b2aa2732d 100644
@@ -39,6 +39,9 @@ struct thread_struct {
 	unsigned long s[12];	/* s[0]: frame pointer */
 	struct __riscv_d_ext_state fstate;
 	unsigned long bad_cause;
+#ifdef CONFIG_RISCV_ISA_SSQOSID
+	u32 sqoscfg;
+#endif
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/qos.h new file +40 -0
new file mode 100644
index 000000000000..297e7fb64d80
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_RISCV_QOS_H
+#define _ASM_RISCV_QOS_H
+
+#ifdef CONFIG_RISCV_ISA_SSQOSID
+
+#include <linux/sched.h>
+#include <linux/jump_label.h>
+
+#include <asm/barrier.h>
+#include <asm/csr.h>
+#include <asm/hwcap.h>
+
+/* cached value of sqoscfg csr for each cpu */
+static DEFINE_PER_CPU(u32, cpu_sqoscfg);
+
+static void __qos_sched_in(struct task_struct *task)
+{
+	u32 *cpu_sqoscfg_ptr = this_cpu_ptr(&cpu_sqoscfg);
+	u32 thread_sqoscfg;
+
+	thread_sqoscfg = READ_ONCE(task->thread.sqoscfg);
+
+	if (thread_sqoscfg != *cpu_sqoscfg_ptr) {
+		*cpu_sqoscfg_ptr = thread_sqoscfg;
+		csr_write(CSR_SQOSCFG, thread_sqoscfg);
+	}
+}
+
+static inline void qos_sched_in(struct task_struct *task)
+{
+	if (riscv_has_extension_likely(RISCV_ISA_EXT_SSQOSID))
+		__qos_sched_in(task);
+}
+#else
+
+static inline void qos_sched_in(struct task_struct *task) {}
+
+#endif /* CONFIG_RISCV_ISA_SSQOSID */
+#endif /* _ASM_RISCV_QOS_H */
arch/riscv/include/asm/switch_to.h +2 -0
index 60f8ca01d36e..75d9bfd766af 100644
@@ -12,6 +12,7 @@
 #include <asm/processor.h>
 #include <asm/ptrace.h>
 #include <asm/csr.h>
+#include <asm/qos.h>
 
 #ifdef CONFIG_FPU
 extern void __fstate_save(struct task_struct *save_to);
@@ -79,6 +80,7 @@ do {							\
 	if (has_fpu())					\
 		__switch_to_aux(__prev, __next);	\
 	((last) = __switch_to(__prev, __next));		\
+	qos_sched_in(__next);				\
 } while (0)
 
 #endif /* _ASM_RISCV_SWITCH_TO_H */
Signature · 7 lines
2.34.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#3 lore raw

Re: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR

kernel test robot noticed the following build errors:

Hi Drew,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on d34a6b715a23ccd9c9d0bc7a475bea59dc3e28b2]

url:    https://github.com/intel-lab-lkp/linux/commits/Drew-Fustini/RISC-V-Detect-the-Ssqosid-extension/20230410-123605
base:   d34a6b715a23ccd9c9d0bc7a475bea59dc3e28b2
patch link:    https://lore.kernel.org/r/20230410043646.3138446-3-dfustini%40baylibre.com
patch subject: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR
config: riscv-randconfig-r015-20230409 (https://download.01.org/0day-ci/archive/20230410/[email protected]/config)
compiler: riscv64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/e4cdc958b7e755dd86202b069497aa36e01b325a
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Drew-Fustini/RISC-V-Detect-the-Ssqosid-extension/20230410-123605 git checkout e4cdc958b7e755dd86202b069497aa36e01b325a
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> riscv64-linux-ld: arch/riscv/kernel/ptrace.o:arch/riscv/include/asm/qos.h:15: multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:arch/riscv/include/asm/qos.h:15: first defined here

riscv64-linux-ld: arch/riscv/kernel/signal.o:arch/riscv/include/asm/qos.h:15: multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:arch/riscv/include/asm/qos.h:15: first defined here
riscv64-linux-ld: kernel/sched/core.o:arch/riscv/include/asm/qos.h:15: multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:arch/riscv/include/asm/qos.h:15: first defined here
riscv64-linux-ld: kernel/sched/fair.o:arch/riscv/include/asm/qos.h:15: multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:arch/riscv/include/asm/qos.h:15: first defined here
riscv64-linux-ld: kernel/sched/build_utility.o:arch/riscv/include/asm/qos.h:15: multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:arch/riscv/include/asm/qos.h:15: first defined here

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#4 lore raw

Re: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR

In reply to Drew Fustini
kernel test robot noticed the following build errors:

Hi Drew,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on d34a6b715a23ccd9c9d0bc7a475bea59dc3e28b2]

url:    https://github.com/intel-lab-lkp/linux/commits/Drew-Fustini/RISC-V-Detect-the-Ssqosid-extension/20230410-123605
base:   d34a6b715a23ccd9c9d0bc7a475bea59dc3e28b2
patch link:    https://lore.kernel.org/r/20230410043646.3138446-3-dfustini%40baylibre.com
patch subject: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR
config: riscv-allmodconfig (https://download.01.org/0day-ci/archive/20230410/[email protected]/config)
compiler: riscv64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/e4cdc958b7e755dd86202b069497aa36e01b325a
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Drew-Fustini/RISC-V-Detect-the-Ssqosid-extension/20230410-123605 git checkout e4cdc958b7e755dd86202b069497aa36e01b325a
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=riscv olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> riscv64-linux-ld: arch/riscv/kernel/ptrace.o:(.discard+0x0): multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:(.discard+0x0): first defined here

riscv64-linux-ld: arch/riscv/kernel/signal.o:(.discard+0x0): multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:(.discard+0x0): first defined here
riscv64-linux-ld: arch/riscv/kernel/compat_signal.o:(.discard+0x0): multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:(.discard+0x0): first defined here
riscv64-linux-ld: kernel/sched/core.o:(.discard+0x5): multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:(.discard+0x0): first defined here
riscv64-linux-ld: kernel/sched/fair.o:(.discard+0x2): multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:(.discard+0x0): first defined here
riscv64-linux-ld: kernel/sched/build_utility.o:(.discard+0x13): multiple definition of `__pcpu_unique_cpu_sqoscfg'; arch/riscv/kernel/process.o:(.discard+0x0): first defined here

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#5 lore raw

Re: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR

In reply to Drew Fustini

Hey Drew,

(Don't get your hopes up, I don't have anything really meaningful to contribute, sorry.)

Quoted 52 lines · Drew Fustini

Add support for the sqoscfg CSR defined in the Ssqosid ISA extension (Supervisor-mode Quality of Service ID). The CSR contains two fields:

- Resource Control ID (RCID) used determine resource allocation
- Monitoring Counter ID (MCID) used to track resource usage

Requests from a hart to shared resources like cache will be tagged with these IDs. This allows the usage of shared resources to be associated with the task currently running on the hart.

A sqoscfg field is added to thread_struct and has the same format as the sqoscfg CSR. This allows the scheduler to set the hart's sqoscfg CSR to contain the RCID and MCID for the task that is being scheduled in. The sqoscfg CSR is only written to if the thread_struct.sqoscfg is different from the current value of the CSR.

A per-cpu variable cpu_sqoscfg is used to mirror that state of the CSR. This is because access to L1D hot memory should be several times faster than a CSR read. Also, in the case of virtualization, accesses to this CSR are trapped in the hypervisor.

---
Note: the Ssqosid extension and CBQRI spec are still in a draft state. The CSR address of sqoscfg is not final.

Changes from the original patch [1]:
- Rebase from 6.0 to 6.3
- Simplify per-cpu variable from struct to u32 with just sqoscfg
- Move qoscfg to thread_struct in arch/riscv/include/asm/processor.h This avoids changing task_struct in /include/linux/sched.h
- Reword commit description
- Reword Kconfig description

[1] https://github.com/rivosinc/linux/commit/8454b793a62be21d39e5826ef5241dfa06198ba9

 arch/riscv/Kconfig                 | 19 ++++++++++++++
 arch/riscv/include/asm/csr.h       |  8 ++++++
 arch/riscv/include/asm/processor.h |  3 +++
 arch/riscv/include/asm/qos.h       | 40 ++++++++++++++++++++++++++++++
 arch/riscv/include/asm/switch_to.h |  2 ++
 5 files changed, 72 insertions(+)
 create mode 100644 arch/riscv/include/asm/qos.h
arch/riscv/Kconfig +0 -0
index cc02eb9eee1f..03f22b7fe34b 100644

arch/riscv/Kconfig config RISCV_ISA_SVNAPOT · line 426

Quoted 9 lines
arch/riscv/Kconfig +6 -0
@@ -418,6 +418,25 @@ config RISCV_ISA_SVNAPOT
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_SSQOSID
+	bool "Ssqosid extension support"
+	default y
+	help
+	  Adds support for the Ssqosid ISA extension (Supervisor-mode
+	  Quality of Service ID).

Could you add "long form" text in brackets here to the bool line, a la:
https://patchwork.kernel.org/project/linux-riscv/patch/20230405-pucker-cogwheel-3a999a94a2f2@wendy/

Quoted 82 lines
arch/riscv/Kconfig +13 -0
+
+	  Ssqosid defines the sqoscfg CSR which allows the system to tag
+	  the running process with RCID (Resource Control ID) and MCID
+	  (Monitoring Counter ID). The RCID is used determine resource
+	  allocation. The MCID is used to track resource usage in event
+	  counters.
+
+	  For example, a cache controller may use the RCID to apply a
+	  cache partitioning scheme and use the MCID to track how much
+	  cache a process, or a group of processes, is using.
+
+	  If you don't know what to do here, say Y.
+
 config RISCV_ISA_SVPBMT
 	bool "SVPBMT extension support"
 	depends on 64BIT && MMU
arch/riscv/include/asm/csr.h +8 -0
index 7c2b8cdb7b77..17d04a0cacd6 100644
@@ -59,6 +59,13 @@
 #define SATP_ASID_MASK	_AC(0xFFFF, UL)
 #endif
 
+/* SQOSCFG fields */
+#define SQOSCFG_RCID_MASK	_AC(0x00000FFF, UL)
+#define SQOSCFG_MCID_MASK	SQOSCFG_RCID_MASK
+#define SQOSCFG_MCID_SHIFT	16
+#define SQOSCFG_MASK		((SQOSCFG_MCID_MASK << SQOSCFG_MCID_SHIFT) | \
+				  SQOSCFG_RCID_MASK)
+
 /* Exception cause high bit - is an interrupt if set */
 #define CAUSE_IRQ_FLAG		(_AC(1, UL) << (__riscv_xlen - 1))
 
@@ -245,6 +252,7 @@
 #define CSR_STVAL		0x143
 #define CSR_SIP			0x144
 #define CSR_SATP		0x180
+#define CSR_SQOSCFG		0x181
 
 #define CSR_STIMECMP		0x14D
 #define CSR_STIMECMPH		0x15D
arch/riscv/include/asm/processor.h +3 -0
index 94a0590c6971..724b2aa2732d 100644
@@ -39,6 +39,9 @@ struct thread_struct {
 	unsigned long s[12];	/* s[0]: frame pointer */
 	struct __riscv_d_ext_state fstate;
 	unsigned long bad_cause;
+#ifdef CONFIG_RISCV_ISA_SSQOSID
+	u32 sqoscfg;
+#endif
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/qos.h new file +20 -0
new file mode 100644
index 000000000000..297e7fb64d80
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_RISCV_QOS_H
+#define _ASM_RISCV_QOS_H
+
+#ifdef CONFIG_RISCV_ISA_SSQOSID
+
+#include <linux/sched.h>
+#include <linux/jump_label.h>
+
+#include <asm/barrier.h>
+#include <asm/csr.h>
+#include <asm/hwcap.h>
+
+/* cached value of sqoscfg csr for each cpu */
+static DEFINE_PER_CPU(u32, cpu_sqoscfg);
+
+static void __qos_sched_in(struct task_struct *task)
+{
+	u32 *cpu_sqoscfg_ptr = this_cpu_ptr(&cpu_sqoscfg);
+	u32 thread_sqoscfg;

arch/riscv/include/asm/qos.h line 30

Quoted 10 lines
arch/riscv/include/asm/qos.h new file +10 -0
+
+	thread_sqoscfg = READ_ONCE(task->thread.sqoscfg);
+
+	if (thread_sqoscfg != *cpu_sqoscfg_ptr) {
+		*cpu_sqoscfg_ptr = thread_sqoscfg;
+		csr_write(CSR_SQOSCFG, thread_sqoscfg);
+	}
+}
+
+static inline void qos_sched_in(struct task_struct *task)

"qos" is a pretty generic prefix, no? Do you think we'd be better off prefixing this (and every other extension related thing) with riscv_?

Quoted 23 lines
arch/riscv/include/asm/qos.h new file +10 -0
+{
+	if (riscv_has_extension_likely(RISCV_ISA_EXT_SSQOSID))
+		__qos_sched_in(task);
+}
+#else
+
+static inline void qos_sched_in(struct task_struct *task) {}
+
+#endif /* CONFIG_RISCV_ISA_SSQOSID */
+#endif /* _ASM_RISCV_QOS_H */
arch/riscv/include/asm/switch_to.h +1 -0
index 60f8ca01d36e..75d9bfd766af 100644
@@ -12,6 +12,7 @@
 #include <asm/processor.h>
 #include <asm/ptrace.h>
 #include <asm/csr.h>
+#include <asm/qos.h>
 
 #ifdef CONFIG_FPU
 extern void __fstate_save(struct task_struct *save_to);

arch/riscv/include/asm/switch_to.h do { \ · line 83

Quoted 4 lines
arch/riscv/include/asm/switch_to.h +1 -0
@@ -79,6 +80,7 @@ do {							\
 	if (has_fpu())					\
 		__switch_to_aux(__prev, __next);	\
 	((last) = __switch_to(__prev, __next));		\
+	qos_sched_in(__next);				\

Both FPU and vector do:

|	if (has_fpu())					\
|		__switch_to_fpu(__prev, __next);	\
|	if (has_vector())				\
|		__switch_to_vector(__prev, __next);	\

Is it just my OCD that wants ssqosid to be the same?
It'd also do away with that seems a bit weird to me: having qos_sched_in() and __qos_sched_in().
Even if you don't make them similar, what's the rationale behind not inverting the extension check & returning early from a single function.

This is kinda above my pay grade, so let me know what I've inevitably missed,
Conor.

Quoted 5 lines
arch/riscv/include/asm/switch_to.h +0 -0
 #endif /* _ASM_RISCV_SWITCH_TO_H */
Signature · 1 line
2.34.1
#6 lore raw

Re: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR

Quoted 4 lines · Conor Dooley

Hey Drew,

(Don't get your hopes up, I don't have anything really meaningful to contribute, sorry.)

Hi, thanks for the feedback...

Quoted 57 lines · Drew Fustini
Quoted 55 lines

Add support for the sqoscfg CSR defined in the Ssqosid ISA extension (Supervisor-mode Quality of Service ID). The CSR contains two fields:

- Resource Control ID (RCID) used determine resource allocation
- Monitoring Counter ID (MCID) used to track resource usage

Requests from a hart to shared resources like cache will be tagged with these IDs. This allows the usage of shared resources to be associated with the task currently running on the hart.

A sqoscfg field is added to thread_struct and has the same format as the sqoscfg CSR. This allows the scheduler to set the hart's sqoscfg CSR to contain the RCID and MCID for the task that is being scheduled in. The sqoscfg CSR is only written to if the thread_struct.sqoscfg is different from the current value of the CSR.

A per-cpu variable cpu_sqoscfg is used to mirror that state of the CSR. This is because access to L1D hot memory should be several times faster than a CSR read. Also, in the case of virtualization, accesses to this CSR are trapped in the hypervisor.

---
Note: the Ssqosid extension and CBQRI spec are still in a draft state. The CSR address of sqoscfg is not final.

Changes from the original patch [1]:
- Rebase from 6.0 to 6.3
- Simplify per-cpu variable from struct to u32 with just sqoscfg
- Move qoscfg to thread_struct in arch/riscv/include/asm/processor.h This avoids changing task_struct in /include/linux/sched.h
- Reword commit description
- Reword Kconfig description

[1] https://github.com/rivosinc/linux/commit/8454b793a62be21d39e5826ef5241dfa06198ba9

 arch/riscv/Kconfig                 | 19 ++++++++++++++
 arch/riscv/include/asm/csr.h       |  8 ++++++
 arch/riscv/include/asm/processor.h |  3 +++
 arch/riscv/include/asm/qos.h       | 40 ++++++++++++++++++++++++++++++
 arch/riscv/include/asm/switch_to.h |  2 ++
 5 files changed, 72 insertions(+)
 create mode 100644 arch/riscv/include/asm/qos.h
arch/riscv/Kconfig +0 -0
index cc02eb9eee1f..03f22b7fe34b 100644
@@ -418,6 +418,25 @@ config RISCV_ISA_SVNAPOT
 
 	  If you don't know what to do here, say Y.
 
Quoted 9 lines

arch/riscv/Kconfig config RISCV_ISA_SVNAPOT · line 426

Quoted 6 lines
arch/riscv/Kconfig +6 -0
+config RISCV_ISA_SSQOSID
+	bool "Ssqosid extension support"
+	default y
+	help
+	  Adds support for the Ssqosid ISA extension (Supervisor-mode
+	  Quality of Service ID).

Could you add "long form" text in brackets here to the bool line, a la:
https://patchwork.kernel.org/project/linux-riscv/patch/20230405-pucker-cogwheel-3a999a94a2f2@wendy/

Thanks, that is a good idea. It would make it easier for the person using menuconfig as they won't have to access the help screen just to see what the acronym stands for.

Quoted 87 lines
Quoted 87 lines
arch/riscv/Kconfig +13 -0
+
+	  Ssqosid defines the sqoscfg CSR which allows the system to tag
+	  the running process with RCID (Resource Control ID) and MCID
+	  (Monitoring Counter ID). The RCID is used determine resource
+	  allocation. The MCID is used to track resource usage in event
+	  counters.
+
+	  For example, a cache controller may use the RCID to apply a
+	  cache partitioning scheme and use the MCID to track how much
+	  cache a process, or a group of processes, is using.
+
+	  If you don't know what to do here, say Y.
+
 config RISCV_ISA_SVPBMT
 	bool "SVPBMT extension support"
 	depends on 64BIT && MMU
arch/riscv/include/asm/csr.h +8 -0
index 7c2b8cdb7b77..17d04a0cacd6 100644
@@ -59,6 +59,13 @@
 #define SATP_ASID_MASK	_AC(0xFFFF, UL)
 #endif
 
+/* SQOSCFG fields */
+#define SQOSCFG_RCID_MASK	_AC(0x00000FFF, UL)
+#define SQOSCFG_MCID_MASK	SQOSCFG_RCID_MASK
+#define SQOSCFG_MCID_SHIFT	16
+#define SQOSCFG_MASK		((SQOSCFG_MCID_MASK << SQOSCFG_MCID_SHIFT) | \
+				  SQOSCFG_RCID_MASK)
+
 /* Exception cause high bit - is an interrupt if set */
 #define CAUSE_IRQ_FLAG		(_AC(1, UL) << (__riscv_xlen - 1))
 
@@ -245,6 +252,7 @@
 #define CSR_STVAL		0x143
 #define CSR_SIP			0x144
 #define CSR_SATP		0x180
+#define CSR_SQOSCFG		0x181
 
 #define CSR_STIMECMP		0x14D
 #define CSR_STIMECMPH		0x15D
arch/riscv/include/asm/processor.h +3 -0
index 94a0590c6971..724b2aa2732d 100644
@@ -39,6 +39,9 @@ struct thread_struct {
 	unsigned long s[12];	/* s[0]: frame pointer */
 	struct __riscv_d_ext_state fstate;
 	unsigned long bad_cause;
+#ifdef CONFIG_RISCV_ISA_SSQOSID
+	u32 sqoscfg;
+#endif
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/qos.h new file +25 -0
new file mode 100644
index 000000000000..297e7fb64d80
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_RISCV_QOS_H
+#define _ASM_RISCV_QOS_H
+
+#ifdef CONFIG_RISCV_ISA_SSQOSID
+
+#include <linux/sched.h>
+#include <linux/jump_label.h>
+
+#include <asm/barrier.h>
+#include <asm/csr.h>
+#include <asm/hwcap.h>
+
+/* cached value of sqoscfg csr for each cpu */
+static DEFINE_PER_CPU(u32, cpu_sqoscfg);
+
+static void __qos_sched_in(struct task_struct *task)
+{
+	u32 *cpu_sqoscfg_ptr = this_cpu_ptr(&cpu_sqoscfg);
+	u32 thread_sqoscfg;
+
+	thread_sqoscfg = READ_ONCE(task->thread.sqoscfg);
+
+	if (thread_sqoscfg != *cpu_sqoscfg_ptr) {
+		*cpu_sqoscfg_ptr = thread_sqoscfg;
Quoted 8 lines

arch/riscv/include/asm/qos.h line 30

Quoted 5 lines
arch/riscv/include/asm/qos.h new file +5 -0
+		csr_write(CSR_SQOSCFG, thread_sqoscfg);
+	}
+}
+
+static inline void qos_sched_in(struct task_struct *task)

"qos" is a pretty generic prefix, no? Do you think we'd be better off prefixing this (and every other extension related thing) with riscv_?

That's a fair point. No harm in adding the 'riscv_' to make the context clearer.

Quoted 30 lines
Quoted 24 lines
arch/riscv/include/asm/qos.h new file +10 -0
+{
+	if (riscv_has_extension_likely(RISCV_ISA_EXT_SSQOSID))
+		__qos_sched_in(task);
+}
+#else
+
+static inline void qos_sched_in(struct task_struct *task) {}
+
+#endif /* CONFIG_RISCV_ISA_SSQOSID */
+#endif /* _ASM_RISCV_QOS_H */
arch/riscv/include/asm/switch_to.h +1 -0
index 60f8ca01d36e..75d9bfd766af 100644
@@ -12,6 +12,7 @@
 #include <asm/processor.h>
 #include <asm/ptrace.h>
 #include <asm/csr.h>
+#include <asm/qos.h>
 
 #ifdef CONFIG_FPU
 extern void __fstate_save(struct task_struct *save_to);

arch/riscv/include/asm/switch_to.h do { \ · line 83

Quoted 4 lines
arch/riscv/include/asm/switch_to.h +1 -0
@@ -79,6 +80,7 @@ do {							\
 	if (has_fpu())					\
 		__switch_to_aux(__prev, __next);	\
 	((last) = __switch_to(__prev, __next));		\
+	qos_sched_in(__next);				\

Both FPU and vector do:

Quoted 6 lines
|	if (has_fpu())					\
|		__switch_to_fpu(__prev, __next);	\
|	if (has_vector())				\
|		__switch_to_vector(__prev, __next);	\

Is it just my OCD that wants ssqosid to be the same?

Thanks, I had not noticed those changes from the vector patch series [1] until you pointed it out. The handling of sqoscfg could be converted to that pattern too.

Quoted 4 lines

It'd also do away with that seems a bit weird to me: having qos_sched_in() and __qos_sched_in().
Even if you don't make them similar, what's the rationale behind not inverting the extension check & returning early from a single function.

The goal was so the inline qos_sched_in() would turn into a nop when Ssqosid extensions not present. If Ssqosid was present, then the "real" function __qos_sched_in() would be called.

However, having looked at the handling of fpu and vector in the vector series, I think will redo the sqoscfg handling to follow that pattern.

thanks,
drew

[1] https://lore.kernel.org/linux-riscv/[email protected]/

#7 lore raw

Re: [RFC PATCH 2/2] RISC-V: Add support for sqoscfg CSR

Quoted 28 lines · Drew Fustini
Quoted 11 lines · Conor Dooley

arch/riscv/include/asm/switch_to.h do { \ · line 83

Quoted 7 lines · Drew Fustini

#ifdef CONFIG_FPU
extern void __fstate_save(struct task_struct *save_to);

arch/riscv/include/asm/switch_to.h +1 -0
@@ -79,6 +80,7 @@ do {							\
 	if (has_fpu())					\
 		__switch_to_aux(__prev, __next);	\
 	((last) = __switch_to(__prev, __next));		\
+	qos_sched_in(__next);				\

Both FPU and vector do:

Quoted 6 lines
|	if (has_fpu())					\
|		__switch_to_fpu(__prev, __next);	\
|	if (has_vector())				\
|		__switch_to_vector(__prev, __next);	\

Is it just my OCD that wants ssqosid to be the same?

Thanks, I had not noticed those changes from the vector patch series [1] until you pointed it out. The handling of sqoscfg could be converted to that pattern too.

Quoted 4 lines

It'd also do away with that seems a bit weird to me: having qos_sched_in() and __qos_sched_in().
Even if you don't make them similar, what's the rationale behind not inverting the extension check & returning early from a single function.

Quoted 6 lines

The goal was so the inline qos_sched_in() would turn into a nop when Ssqosid extensions not present. If Ssqosid was present, then the "real" function __qos_sched_in() would be called.

However, having looked at the handling of fpu and vector in the vector series, I think will redo the sqoscfg handling to follow that pattern.

Aye. With the way they do it, if !CONFIG_..._SSQOSID, DCE should remove the function call entirely & the has_extension_[un]likely() stuff will give you your nop for kernels with CONFIG_..._SSQOSID but no hardware support.

I think it looks "nicer"/more obvious if you copy what the fpu/vector stuff does /shrug

Cheers,
Conor.