Skip to content

RFC riscv: implement Ssqosid extension and CBQRI controllers

Drew Fustini qemu-devel qemu-riscv 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 RFC Latest9 patches 1 replyIn review
  2. v1 RFC8 patches 1 replySuperseded

Changes in v2 from the cover letter

----------------
- Rebase on current master (8.0.50) instead of 8.0.0-rc4
- Configure CBQRI controllers based on device properties in command line arguments instead of a fixed example SoC configuration.
- Move TYPE_RISCV_CBQRI_BC and TYPE_RISCV_CBQRI_CC into header so that machines may use it (suggested by Alistair).
- Change 'select RISC_CBQRI' to 'imply RISCV_CBQRI' for RISCV_VIRT.
- Patches 8/9 could be dropped as they are not needed for the CBQRI proof-of-concept to work. They are only meant to serve as an example for those implementing new machines.

[1] https://github.com/riscv-non-isa/riscv-cbqri/blob/main/riscv-cbqri.pdf
[2] https://gitlab.baylibre.com/baylibre/qemu/-/tree/riscv-cbqri-rfc-v2
[3] https://lists.riscv.org/g/tech-cbqri/message/38
[4] https://lore.kernel.org/linux-riscv/[email protected]/
[5] https://lore.kernel.org/linux-riscv/[email protected]/

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9riscv: implement Ssqosid extension and sqoscfg CSRdisas/riscv.c target/riscv/cpu.c target/riscv/cpu.h +2 more+45 -01 reply
2/9hw/riscv: define capabilities of CBQRI controllersinclude/hw/riscv/cbqri.h+81 -00 replies
3/9hw/riscv: implement CBQRI capacity controllerhw/riscv/cbqri_capacity.c+614 -00 replies
4/9hw/riscv: implement CBQRI bandwidth controllerhw/riscv/cbqri_bandwidth.c+600 -00 replies
5/9hw/riscv: Kconfig: add CBQRI optionshw/riscv/Kconfig+4 -00 replies
6/9hw/riscv: meson: add CBQRI controllers to the buildhw/riscv/meson.build+2 -00 replies
7/9hw/riscv: add CBQRI controllers to virt machinehw/riscv/virt.c+3 -00 replies
8/9hw/riscv: instantiate CBQRI controllers for an example SoChw/riscv/cbqri_example_soc.c include/hw/riscv/cbqri.h+126 -00 replies
9/9hw/riscv: build example SoC when CBQRI_EXAMPLE_SOC enabledhw/riscv/Kconfig hw/riscv/meson.build+6 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [RFC PATCH v2 0/9] riscv: implement Ssqosid extension and CBQRI controllers

This RFC series implements the Ssqosid extension and the sqoscfg CSR as defined in the RISC-V Capacity and Bandwidth Controller QoS Register Interface (CBQRI) specification [1]. Quality of Service (QoS) in this context is concerned with shared resources on an SoC such as cache capacity and memory bandwidth.

These patches are available as a public git branch [2].

sqoscfg CSR
-----------
The sqoscfg CSR provides a mechanism by which a software workload (e.g. a process or a set of processes) can be associated with a resource control ID (RCID) and a monitoring counter ID (MCID) that accompanies each request made by the hart to shared resources like cache.

CBQRI defines operations to configure resource usage limits, in the form of capacity or bandwidth, for an RCID. CBQRI also defines operations to configure counters to track the resource utilization per MCID.

CBQRI controllers
-----------------
This series also implements an CBQRI capacity controller and an CBQRI bandwidth controller which can be configured from the command line:

  $ qemu-system-riscv64 -M virt ... \
      -device riscv.cbqri.capacity,mmio_base=0x04828000[,...] \
      -device riscv.cbqri.bandwidth,mmio_base=0x04829000[,...]

The mmio_base option is mandatory, the others are optional.

As many -device arguments as wanted can be provided as long as their mmio regions don't conflict.

To see all possible options:

  $ qemu-system-riscv64 -device riscv.cbqri.capacity,help
  riscv.cbqri.capacity options:
    alloc_op_config_limit=<bool> -  (default: true)
    alloc_op_flush_rcid=<bool> -  (default: true)
    alloc_op_read_limit=<bool> -  (default: true)
    at_code=<bool>         -  (default: true)
    at_data=<bool>         -  (default: true)
    max_mcids=<uint16>     -  (default: 256)
    max_rcids=<uint16>     -  (default: 64)   
    mmio_base=<uint64>     -  (default: 0)
    mon_evt_id_none=<bool> -  (default: true)
    mon_evt_id_occupancy=<bool> -  (default: true)
    mon_op_config_event=<bool> -  (default: true)
    mon_op_read_counter=<bool> -  (default: true)
    ncblks=<uint16>        -  (default: 16)
    target=<str> 
  $ qemu-system-riscv64 -device riscv.cbqri.bandwidth,help
  riscv.cbqri.bandwidth options:
    alloc_op_config_limit=<bool> -  (default: true)
    alloc_op_read_limit=<bool> -  (default: true)
    at_code=<bool>         -  (default: true)
    at_data=<bool>         -  (default: true)
    max_mcids=<uint16>     -  (default: 256)
    max_rcids=<uint16>     -  (default: 64)
    mmio_base=<uint64>     -  (default: 0)
    mon_evt_id_none=<bool> -  (default: true)
    mon_evt_id_rdonly_count=<bool> -  (default: true)
    mon_evt_id_rdwr_count=<bool> -  (default: true)
    mon_evt_id_wronly_count=<bool> -  (default: true)
    mon_op_config_event=<bool> -  (default: true)
    mon_op_read_counter=<bool> -  (default: true)
    nbwblks=<uint16>       -  (default: 1024)
    target=<str>

Boolean options correspond to hardware capabilities that can be disabled

CBQRI proof-of-concept
----------------------
A hypothetical SoC with the following CBQRI controller configuration is used for the CBQRI proof-of-concept:

- L2 cache controllers
- Resource type: Capacity
- Number of capacity blocks (NCBLKS): 12
- In the context of a set-associative cache, the number of capacity blocks can be thought of as the number of ways
- Number of access types: 2 (code and data)
- Usage monitoring not supported
- Capacity allocation operations: CONFIG_LIMIT, READ_LIMIT

- Last-level cache (LLC) controller
- Resource type: Capacity
- Number of capacity blocks (NCBLKS): 16
- Number of access types: 2 (code and data)
- Usage monitoring operations: CONFIG_EVENT, READ_COUNTER
- Event IDs supported: None, Occupancy
- Capacity allocation ops: CONFIG_LIMIT, READ_LIMIT, FLUSH_RCID

- Memory controllers
- Resource type: Bandwidth
- Number of bandwidth blocks (NBWBLKS): 1024
- Bandwidth blocks do not have a unit but instead represent a portion of the total bandwidth resource. For NWBLKS of 1024, each block represents about 0.1% of the bandwidth resource.
- Maximum reserved bandwidth blocks (MRBWB): 819 [80% of NBWBLKS]
- Number of access types: 1 (no code/data differentiation)
- Usage monitoring operations: CONFIG_EVENT, READ_COUNTER
- Event IDs supported: None, Total read/write byte count, Total read byte count, Total write byte count
- Bandwidth allocation operations: CONFIG_LIMIT, READ_LIMIT

The memory map used for the proof-of-concept:

  Base addr  Size
  0x4820000  4KB  Cluster 0 L2 cache controller
  0x4821000  4KB  Cluster 1 L2 cache controller
  0x4828000  4KB  Memory controller 0
  0x4829000  4KB  Memory controller 1
  0x482a000  4KB  Memory controller 2
  0x482b000  4KB  Shared LLC cache controller

This configuration is meant to provide a "concrete" example for software (like Linux) to test against. It represents just one of many possible ways for hardware to implement the CBQRI spec.

The CBQRI proof-of-concept configuration is created with the following:

  qemu-system-riscv64 \
        -M virt \
        -nographic \
        -smp 8 \
        -device riscv.cbqri.capacity,max_mcids=256,max_rcids=64,ncblks=12,alloc_op_flush_rcid=false,mon_op_config_event=false,mon_op_read_counter=false,mon_evt_id_none=false,mon_evt_id_occupancy=false,mmio_base=0x04820000 \
        -device riscv.cbqri.capacity,max_mcids=256,max_rcids=64,ncblks=12,alloc_op_flush_rcid=false,mon_op_config_event=false,mon_op_read_counter=false,mon_evt_id_none=false,mon_evt_id_occupancy=false,mmio_base=0x04821000 \
        -device riscv.cbqri.capacity,max_mcids=256,max_rcids=64,ncblks=16,mmio_base=0x0482B000 \
        -device riscv.cbqri.bandwidth,max_mcids=256,max_rcids=64,nbwblks=1024,mrbwb=819,mmio_base=0x04828000 \
        -device riscv.cbqri.bandwidth,max_mcids=256,max_rcids=64,nbwblks=1024,mrbwb=819,mmio_base=0x04829000 \
        -device riscv.cbqri.bandwidth,max_mcids=256,max_rcids=64,nbwblks=1024,mrbwb=819,mmio_base=0x0482a000

In addition, please note that this RFC series only implements the register interface that CBQRI specifies. It does not attempt to emulate the performance impact of configuring limits on shared resources like cache and memory bandwidth. Similarly, the code does not attempt to emulate cache and memory bandwidth utilization, like what would be observed on a real hardware system implementing CBQRI.

Status of CBQRI
---------------
The CBQRI spec is still in a draft state and is undergoing review [3]. It is possible there will be changes to the Ssqosid extension and the CBQRI spec. For example, the sqoscfg CSR address is not yet finalized.

The goal of this Qemu patch series, along with complimentary Linux patch series [4][5], is to satisfy the software proof of concept requirement or CBQRI to be frozen.

Future work
-----------
In the future, the device tree generation will be expanded to include CBQRI-related properties. Currently, these are added to the dumped dtb that Linux consumes for the CBQRI proof-of-concept [5].

Changes since v1
----------------
- Rebase on current master (8.0.50) instead of 8.0.0-rc4
- Configure CBQRI controllers based on device properties in command line arguments instead of a fixed example SoC configuration.
- Move TYPE_RISCV_CBQRI_BC and TYPE_RISCV_CBQRI_CC into header so that machines may use it (suggested by Alistair).
- Change 'select RISC_CBQRI' to 'imply RISCV_CBQRI' for RISCV_VIRT.
- Patches 8/9 could be dropped as they are not needed for the CBQRI proof-of-concept to work. They are only meant to serve as an example for those implementing new machines.

[1] https://github.com/riscv-non-isa/riscv-cbqri/blob/main/riscv-cbqri.pdf
[2] https://gitlab.baylibre.com/baylibre/qemu/-/tree/riscv-cbqri-rfc-v2
[3] https://lists.riscv.org/g/tech-cbqri/message/38
[4] https://lore.kernel.org/linux-riscv/[email protected]/
[5] https://lore.kernel.org/linux-riscv/[email protected]/