ioctl based CAT interface

Attached is an early version of the ioctl based CAT interface we have been working on.

NOTE: it does not compile, there is no locking, but should be sufficient for interested people to comment.

index db3622f..293726b 100644
@@ -757,6 +757,14 @@ config HPET_EMULATE_RTC
 	def_bool y
 	depends on HPET_TIMER && (RTC=y || RTC=m || RTC_DRV_CMOS=m || RTC_DRV_CMOS=y)
 
+config CACHE_RESERVATION
+	tristate "Cache Reservation Support"
+	default n
+	---help---
+	  This feature makes use of Intel's Cache Allocation Technology to allow the
+	  reservation of portions of the L3 cache to specific tasks. Please, see
+	  Documentation/x86/cache-reservation.txt for more information.
+
 config APB_TIMER
        def_bool y if X86_INTEL_MID
        prompt "Intel MID APB Timer Support" if X86_INTEL_MID

index b1b78ff..57129d6 100644
@@ -110,6 +110,8 @@ obj-$(CONFIG_EFI)			+= sysfb_efi.o
 obj-$(CONFIG_PERF_EVENTS)		+= perf_regs.o
 obj-$(CONFIG_TRACING)			+= tracepoint.o
 
+obj-$(CONFIG_CACHE_RESERVATION)	+= cat/ 
+
 ###
 # 64 bit specific files
 ifeq ($(CONFIG_X86_64),y)

new file mode 100644
index 0000000..031fd64
@@ -0,0 +1 @@
+obj-y += cache_reservation.o

NAK on that without cpuset integration.

I do not want freely migratable tasks having radically different performance profiles depending on which CPU they land.

Please expand on what "cpuset integration" means, operationally. I hope it does not mean "i prefer cgroups as an interface", because that does not mean much to me.

So you are saying this should be based on cgroups? Have you seen the cgroups proposal and the issues with it, that have been posted?

Subject: cat cgroup interface proposal (non hierarchical) was Re: [PATCH V15 00/11] x86: Intel Cache
Allocation Technology Support

https://lkml.org/lkml/2015/11/2/700

I've really no idea what you're trying to say there. That just doesn't parse.

I've posted this internally at people understood and commented on it.

What part you don't understand?

Its a proposal for a cgroups based interface. The description starts from the main mount point of the cgroup. So you would:

#  mount -t cgroup -ointel_cat intel_cat /sys/fs/cgroup/cat/
#  cd /sys/fs/cgroup/cat/
#  ls
cat_hw_info
# mkdir cache_reservation_for_forwarding_app
# ls
cat_hw_info   cache_reservation_for_forwarding_app
# cd cache_reservation_for_forwarding_app
# ls
type	size	socketmask	enable_reservation
# ...

Yes?

cgroup CAT interface (non hierarchical):
---------------------------------------
0) Main directory files:

cat_hw_info
-----------
CAT HW information: CBM length, CDP supported, etc. Information reported per-socket, as sockets can have different configurations. Perhaps should be inside sysfs.
1) Sub-directories represent cache reservations (size,type).

mkdir cache_reservation_for_forwarding_app
cd cache_reservation_for_forwarding_app
echo "80KB" > size
echo "data_and_code" > type
echo "socketmask=0xfffff" > socketmask (optional)
echo "1" > enable_reservation
echo "pid-of-forwarding-main-thread pid-of-helper-thread ..." > tasks Files:

type
----------------
{data_and_code, only_code, only_data}. Type of L3 CAT cache allocation to use. only_code,only_data only supported on CDP capable processors.
size
----
size of L3 cache reservation.
rounding
--------
{round_down,round_up} whether to round up / round down allocation size in kbytes, to cache-way size.
Default: round_up

socketmask
----------
Mask of sockets where the reservation is in effect. A zero bit means the task will not have the L3 cache portion that the cgroup references reserved on that socket.
Default: all sockets set.
enable
------
Allocate reservation with parameters set above. When a reservation is enabled, it reserves L3 cache space on any socket thats specified in "socketmask".

After cgroup has been enabled by a write of "1" to "enable_reservation" file, only the "tasks" file can be modified. To change the size of a cgroup reservation, recreate the directory.

tasks
-----
Contains the list of tasks which use this cache reservation.

Error reporting
---------------
Errors are reported in response to write as appropriate:
for example, write 1 > enable when there is not enough space for "socketmask" would return -ENOSPC, etc.
Write to "enable" without size being set would return -EINVAL, etc. Listing
-------
To list which reservations are in place, search for subdirectories where "enabled" file has value 1.
Semantics: A task has guaranteed cache reservation on any CPU where its scheduled in, for the lifetime of the cgroup, as long as that task is not attached to further cgroups.

That is, a task belonging to cgroup-A can have its cache reservation invalidated when attached to cgroup-B, (reasoning: it might be necessary to reallocate the CBMs to respect contiguous bits in cache, a restriction of the CAT HW interface).

-------
BLOCKER
-------
Can't use cgroups for CAT because:

"Control Groups extends the kernel as follows:

- Each task in the system has a reference-counted pointer to a css_set.

- A css_set contains a set of reference-counted pointers to cgroup_subsys_state objects, one for each cgroup subsystem registered in the system."

You need a task to be part of two cgroups at one time, to support the following configuration:

Task-A: 70% of cache reserved exclusively (reservation-0).
20% of cache reserved (reservation-1).

Task-B: 20% of cache reserved (reservation-1).

Unless reservations are created separately, then added to cgroups:

mount -t cgroup ... /../catcgroup/
cd /../catcgroup/

# create reservations
cd reservations
mkdir reservation-1
echo "80K" > size
echo "socketmask" > ...
echo "1" > enable
mkdir reservation-2 
echo "160K" > size
echo "socketmask" > ...
echo "1" > enable
# attach reservation to cgroups
cd /../catcgroup/
mkdir cgroup-for-threaded-app
echo reservation-1 reservation-2 > reservations
echo "mainthread" > tasks
cd ..
mkdir cgroup-for-helper-thread
echo reservation-1 > reservations
echo "helperthread" > tasks
cd .. 
This way mainthread and helperthread can share "reservation-1".

Why would such a configuration be desired?

Thanks,

tglx

The HW supports cache sharing, its a HW feature.

Makes sense to share reserved cache between threads of an application that for example share a library.

That's a fail right there. Cgroup thingies must be hierarchical.

Why?

What is the problem with ioctls again? (haven't seen any technical argument).

If the ioctls were replaced by say, syscalls you would be happy?

If the ioctls were replaced by attach/detach/delete/unassign commands via directories, would you be happy?

I guess that what Peter is saying is that we don't want tasks attached to a reservation landing on a CPU where the reservation might be different or not existent at all.

Peter, what about integrating this with affinity masks instead of cpusets (I have no idea how cpusets are implemented, but I guess they are a superset of affinity masks).

This way, the ATTACH_RESERVATION command would fail if any of the CPUs in the cpumask are not part of the reservation. And then our code would have to be notified any time the process' affinity mask is changed (we either fail the affinity change or detach the process automatically from the reservation). Does this sound like a good solution?

Then you pin the tasks to not land on those CPUs, explicitly.

If you create a reservation on the CPU, then there is a reservation there. If you don't, then there will be no reservation on that CPU.

The amount of L3 cache you reserve is the same, on each CPU, just the location of L3 cache is different, but the quantity is the same.

(this in case the region shared with HW is different). If the region shared with HW is the same on each socket, then the location (as specified in the capacity bitmask) is the same on every socket. HW limitation.

Yes, perhaps notifications to cpusets is what Peter refers to?

Correct.

No. We're not going to have random drivers muck about with affinity masks, and most certainly not some manky ioctl.

Ok, so, configuration:

Socket-1 Socket-2

pinned thread-A with 100% L3 free
80% of L3
reserved

So it is a problem if a thread running on socket-2 is scheduled to socket-1 because performance is radically different, fine.

Then one way to avoid that is to not allow freely migratable tasks to move to Socket-1. Fine.

Then you want to use cpusets for that.

Can you fill in the blanks what is missing here?

I'm still not seeing what the problem with CAT-cgroup is.

/cgroups/cpuset/
socket-1/cpus = $socket-1
tasks = $thread-A

socket-2/cpus = $socket-2
tasks = $thread-B

/cgroups/cat/
group-A/bitmap = 0x3F / 0xFF
group-A/tasks = $thread-A

group-B/bitmap = 0xFF / 0xFF
group-B/tasks = $thread-B

That gets you thread-A on socket-1 with 6/8 of the L3 and thread-B on socket-2 with 8/8 of the L3.

- need bitmasks per socket (optionally).
- format kept in kernel is not universal (have to convert every time
L3 cache size changes).
- need to specify type (i-cache or d-cache, differentiation supported on newer processors), ok can add more bitmaps.
- position in bitmask represents nothing other than identification of reservation and size, so:

		group-A = 0x3F, group-B = 0xFF
  is the same as
		group-A = 0xFC, group-B = 0xFF

- have to locate a free region every time in the bitmasks. So userspace has to do:

# lock write access to /cgroups/cat/
create group-C, taking into account bitmasks of 
group-A and group-B.
# unlock write access to /cgroups/cat.

But OK, it works, lets use that.

No, the position very much matters; maybe not in this example, but it does the moment you get overlapping bitmaps.

Picking which bits overlap determines which other groups are affected.

This is why a bitmap is more expressive than a single percentage.

Peter, I'm giving a serious try on the cgroups patches and would be glad to be enlightened if I'm missing something. But I don't see how what you're proposing would solve the problem.

My understanding of CAT is that if I want to reserve 80% of the cache in socket-1 to $thread-A I also have to:

1. Create another mask reserving 20% of the cache in socket-1
2. Assign that mask to all other threads that may run in socket-1

If I'm right about this, then when a task with 20% reservation migrates to socket-2 it will only access 20% of the cache there even though there should be no restrictions in socket-2's cache.

If the solution you're proposing means that I should assign all other tasks to $thread-B, then what you're actually doing is pinning all tasks but $thread-A to socket-2. You can do this today without CAT.

Uh what? Task-A was bound to socket-1, it will never get to socket-2.

Clearly I'm not getting these examples you're throwing around.

Also, I explicitly do not want tasks that can migrate between sockets to have different performance profiles across those sockets.

Sure, but you're going to allow other threads besides Task-A to execute in socket-1 too, right? In this case, my understanding of CAT is that those threads will require a mask to restrict them to only 20% of the cache.

It's the "Isolated Bitmask" example from the Intel Manual p. 616 (You'll find the excerpt below, but Figure 17-27 is easier to understand).

My understanding of this example is that in order to isolate a portion of the cache to Task-A I also have to create a second mask which excludes bits reserved to Task-A. This second mask is assigned to all other tasks that will share the L3 cache with Task-A.

"""
Figure 17-27 also shows three examples of sets of Cache Capacity Bitmasks. For simplicity these are represented as 8-bit vectors, though this may vary depending on the implementation and how the mask is mapped to the avail- able cache capacity. The first example shows the default case where all 4 Classes of Service (the total number of COS are implementation-dependent) have full access to the cache. The second case shows an overlapped case, which would allow some lower-priority threads share cache space with the highest priority threads. The third case shows various non-overlapped partitioning schemes. As a matter of software policy for extensibility COS0 should typically be considered and configured as the highest priority COS, followed by COS1, and so on, though there is no hardware restriction enforcing this mapping. When the system boots all threads are initialized to COS0, which has full access to the cache by default.
"""

I think we can solve this problem with the ioctl interface, if that's what you mean.

Going that route, might as well expose the region shared with HW to userspace and let userspace handle the problem of contiguous free regions, which means the cgroups bitmask maps one-to-one to HW bitmap.

All is necessary then is to modify the Intel patches to

1) Support bitmaps per socket.
2) Remove hierarchical support.

3) Lazy enforcement (which can be done later as an improvement).

Consider the following scenario, one server with two sockets:

	socket-1 			socket-2

[ [***] ] [ [***] ]
L3 cache bitmap L3 cache bitmap

[*] refers to the region shared with HW, as reported by CPUID (read the Intel documentation).

socket-1.shared_region_with_hw = [bit 2, bit 5]
socket-2.shared_region_with_hw = [bit 16, bit 18]

Given that your application is critical, you do not want it to share any reservation with HW. I was informed that there is no guarantee these regions end up in the same location for different sockets. Lets say you need 15 bits of reservation, and the total is 20 bits. One possibility would be:

socket-1.reservation = [bit 5, bit 15]
socket-2.reservation = [bit 1, bit 15]

For the current Intel CAT patchset, this restriction exists:

static int cbm_validate_rdt_cgroup(struct intel_rdt *ir, unsigned long cbmvalue)

{
        struct cgroup_subsys_state *css;
        struct intel_rdt *par, *c;
        unsigned long cbm_tmp = 0;
        int err = 0;

        if (!cbm_validate(cbmvalue)) {
                err = -EINVAL;
                goto out_err;
        }

        par = parent_rdt(ir);
        clos_cbm_table_read(par->closid, &cbm_tmp);
        if (!bitmap_subset(&cbmvalue, &cbm_tmp, MAX_CBM_LENGTH)) {
                err = -EINVAL;
                goto out_err;
        }

Do you (or the author of the patch), can explain why is this restriction here?

If the restriction has to be maintained, than one hierarchy per-socket will be necessary to support different bitmaps per socket.

If the restriction can be removed, then non hierarchical support could look like:

/cgroups/cat/group-A/tasks = $thread-A /cgroups/cat/group-A/socket-1/bitmap = 0x3F / 0xFF /cgroups/cat/group-A/socket-2/bitmap = 0x... / 0xFF

Or one l3_cbm file containing one mask per socket, separated by commas, similar to

/sys/devices/system/node/node0/cpumap

There is nothing hierarchical in CAT, its flat.

Each set of tasks is associated with a number of bits in each socket's L3 CBM mask.

OK, probably not cgroups interface (which can't be done unless someone solves the issue of one task on multiple cgroups, or explains why it is not an issue).

So you come with a cpuset configuration:

A cpuset defines a list of CPUs and memory nodes. The CPUs of
a system include all the logical processing units on which a process can execute, including, if present, multiple processor cores within a package and Hyper-Threads within a processor core. Memory nodes include all distinct banks of main memory; small and SMP systems typi‐ cally have just one memory node that contains all the system's main memory, while NUMA (non-uniform memory access) systems have multiple memory nodes.

----

Then for each task in the cpuset, you can configure via priorities the percentage of time each task is allowed to run on the CPUs of the cpuset.

You want something to automatically remove CPUs from cpusets if a given amount of L3 cache is reserved?

Can do that in userspace, no problem.

A task can be part of every controller once, but there is no reason it needs to be the 'same' cgroup for each controller.

That's how cgroups work, always have.

Why would that be a problem? Just don't co-mount them.

We're also writing a user-space app that can manage reservations from user-space. It's integrated with util-linux. I'll setup a repo shortly to share the source code, but here are some examples:

Create a global 2MB reservation:

  # cacheset --create 2M

Create a 2MB reservation on CPUs 0 and 1:

  # cacheset --create 2M --cpu-list 0,1
  # cacheset --create 2M 0x3

Attach pid 4901 to reservation ID 1

  # cacheset --attach 4901 1

Detach pid 4901 to reservation ID 1

  # cacheset --detach 4901 1

Delete reservation ID 1

  # cacheset --delete 1

Hi!

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html