Skip to contentSkip to conversation

perf: Add CPU hotplug support for events

PATCH In review Mukesh Ojha 2 participants 5 messages Aug 1, 2019 – Sep 24, 2019 perf events

lore
Share & tools
lore
#0 lore raw

v5 cover perf: Add CPU hotplug support for events

The embedded world, specifically Android mobile SoCs, rely on CPU hotplugs to manage power and thermal constraints. These hotplugs can happen at a very rapid pace. Adjacently, they also relies on many perf event counters for its management. Therefore, there is a need to preserve these events across hotplugs.

In such a scenario, a perf client (kernel or user-space) can create events even when the CPU is offline. If the CPU comes online during the lifetime of the event, the registered event can start counting spontaneously. As an extension to this, the events' count can also be preserved across CPU hotplugs. This takes the burden off of the clients to monitor the state of the CPU.

The tests were conducted on arm64 device.
/* CPU-1 is offline: Event created when CPU1 is offline */

/ # cat /sys/devices/system/cpu/cpu1/online
1
/ # echo 0 > /sys/devices/system/cpu/cpu1/online

Test used for testing
#!/bin/sh

chmod +x *

# Count the cycles events on cpu-1 for every 200 ms ./perf stat -e cycles -I 200 -C 1 &

# Make the CPU-1 offline and online continuously while true; do
sleep 2
echo 0 > /sys/devices/system/cpu/cpu1/online sleep 2
echo 1 > /sys/devices/system/cpu/cpu1/online
done

Results:
/ # ./test.sh
# time counts unit events

     0.200145885      <not counted>      cycles
     0.410115208      <not counted>      cycles
     0.619922551      <not counted>      cycles
     0.829904635      <not counted>      cycles
     1.039751614      <not counted>      cycles
     1.249547603      <not counted>      cycles
     1.459228280      <not counted>      cycles
     1.665606561      <not counted>      cycles
     1.874981926      <not counted>      cycles
     2.084297811      <not counted>      cycles
     2.293471249      <not counted>      cycles
     2.503231561      <not counted>      cycles
     2.712993332      <not counted>      cycles
     2.922744478      <not counted>      cycles
     3.132502186      <not counted>      cycles
     3.342255050      <not counted>      cycles
     3.552010102      <not counted>      cycles
     3.761760363      <not counted>      cycles
    /* CPU-1 made online: Event started counting */

     3.971459269            1925429      cycles
     4.181325206           19391145      cycles
     4.391074164             113894      cycles
     4.599130519            3150152      cycles
     4.805564737             487122      cycles
     5.015164581             247533      cycles
     5.224764529             103622      cycles
#           time             counts unit events
     5.434360831             238179      cycles
     5.645293799             238895      cycles
     5.854909320             367543      cycles
     6.064487966            2383428      cycles

     /* CPU-1 made offline: counting stopped

     6.274289476      <not counted>      cycles
     6.483493903      <not counted>      cycles
     6.693202705      <not counted>      cycles
     6.902956195      <not counted>      cycles
     7.112714268      <not counted>      cycles
     7.322465570      <not counted>      cycles
     7.532222340      <not counted>      cycles
     7.741975830      <not counted>      cycles
     7.951686246      <not counted>      cycles

    /* CPU-1 made online: Event started counting

     8.161469892           22040750      cycles
     8.371219528             114977      cycles
     8.580979111             259952      cycles
     8.790757132             444661      cycles
     9.000559215             248512      cycles
     9.210385256             246590      cycles
     9.420187704             243819      cycles
     9.630052287            7102438      cycles
     9.839848225             337454      cycles
    10.049645048             644072      cycles
    10.259476246            1855410      cycles
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Alexei Starovoitov <[email protected]>

Mukesh Ojha (1):
perf: event preserve and create across cpu hotplug

 include/linux/perf_event.h |   1 +
 kernel/events/core.c       | 122 +++++++++++++++++++++++++++++----------------
 2 files changed, 79 insertions(+), 44 deletions(-)
Signature · 2 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
#1 lore raw

PATCH v5 1/1 perf: event preserve and create across cpu hotplug

Perf framework doesn't allow preserving CPU events across CPU hotplugs. The events are scheduled out as and when the CPU walks offline. Moreover, the framework also doesn't allow the clients to create events on an offline CPU. As a result, the clients have to keep on monitoring the CPU state until it comes back online.

Therefore, introducing the perf framework to support creation and preserving of (CPU) events for offline CPUs. Through this, the CPU's online state would be transparent to the client and it not have to worry about monitoring the CPU's state. Success would be returned to the client even while creating the event on an offline CPU. If during the lifetime of the event the CPU walks offline, the event would be preserved and would continue to count as soon as (and if) the CPU comes back online.

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Alexei Starovoitov <[email protected]>

---
Change in V5:
=============
- Rebased it.

Change in V4:
=============
- Released, __get_cpu_context would not be correct way to get the cpu context of the cpu which is offline, instead use container_of to get the cpuctx from ctx.

- Changed the goto label name inside event_function_call from 'remove_event_from_context' to 'out'.

Change in V3:
=============
- Jiri has tried perf stat -a and removed one of the cpu from the other terminal. This resulted in a crash. Crash was because in event_function_call(), we were passing NULL as cpuctx in func(event, NULL, ctx, data).Fixed it in this patch.

Change in V2:
=============
As per long back discussion happened at
https://lkml.org/lkml/2018/2/15/1324

Peter.Z. has suggested to do thing in different way and shared patch as well. This patch fixes the issue seen while trying to achieve the purpose.

Fixed issue on top of Peter's patch:
===================================
1. Added a NULL check on task to avoid crash in __perf_install_in_context.

2. while trying to add event to context when cpu is offline. Inside add_event_to_ctx() to make consistent state machine while hotplug.

-event->state += PERF_EVENT_STATE_HOTPLUG_OFFSET;
+event->state = PERF_EVENT_STATE_HOTPLUG_OFFSET;

3. In event_function_call(), added a label 'remove_event_from_ctx' to delete events from context list while cpu is offline.

 include/linux/perf_event.h |   1 +
 kernel/events/core.c       | 123 ++++++++++++++++++++++++++++-----------------
 2 files changed, 79 insertions(+), 45 deletions(-)
include/linux/perf_event.h +1 -0
index 3dc01cf..52b14b2 100644
@@ -511,6 +511,7 @@ enum perf_event_state {
 	PERF_EVENT_STATE_OFF		= -1,
 	PERF_EVENT_STATE_INACTIVE	=  0,
 	PERF_EVENT_STATE_ACTIVE		=  1,
+	PERF_EVENT_STATE_HOTPLUG_OFFSET	= -32,
 };
 
 struct file;
kernel/events/core.c +78 -45

219 lines not on this page · raw message

Signature · 2 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
#2 lore raw

Re: [PATCH V5 1/1] perf: event preserve and create across cpu hotplug

Quoted 26 lines · Mukesh Ojha

Perf framework doesn't allow preserving CPU events across CPU hotplugs. The events are scheduled out as and when the CPU walks offline. Moreover, the framework also doesn't allow the clients to create events on an offline CPU. As a result, the clients have to keep on monitoring the CPU state until it comes back online.

Therefore, introducing the perf framework to support creation and preserving of (CPU) events for offline CPUs. Through this, the CPU's online state would be transparent to the client and it not have to worry about monitoring the CPU's state. Success would be returned to the client even while creating the event on an offline CPU. If during the lifetime of the event the CPU walks offline, the event would be preserved and would continue to count as soon as (and if) the CPU comes back online.

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Quoted 4 lines

---
Change in V5:
=============
- Rebased it.

note that we might need to change how we store cpu topology, now that it can change during the sampling.. like below it's the comparison of header data with and without cpu 1

I think some of the report code checks on topology or caches and it might get confused

perhaps we could watch cpu topology in record and update the data as we see it changing.. future TODO list ;-)

perf stat is probably fine

jirka

---
-# nrcpus online : 39
+# nrcpus online : 40

 # nrcpus avail : 40
 # cpudesc : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
 # cpuid : GenuineIntel,6,85,4
...
 # sibling sockets : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling sockets : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling sockets : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
 # sibling dies    : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling dies    : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling dies    : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
 # sibling threads : 0,20
+# sibling threads : 1,21
 # sibling threads : 2,22
 # sibling threads : 3,23
 # sibling threads : 4,24
@@ -38,9 +39,8 @@
 # sibling threads : 17,37
 # sibling threads : 18,38
 # sibling threads : 19,39
-# sibling threads : 21
 # CPU 0: Core ID 0, Die ID 0, Socket ID 0
-# CPU 1: Core ID -1, Die ID -1, Socket ID -1
+# CPU 1: Core ID 0, Die ID 0, Socket ID 1
 # CPU 2: Core ID 4, Die ID 0, Socket ID 0
 # CPU 3: Core ID 4, Die ID 0, Socket ID 1
 # CPU 4: Core ID 1, Die ID 0, Socket ID 0
@@ -79,14 +79,16 @@
 # CPU 37: Core ID 9, Die ID 0, Socket ID 1
 # CPU 38: Core ID 10, Die ID 0, Socket ID 0
 # CPU 39: Core ID 10, Die ID 0, Socket ID 1
-# node0 meminfo  : total = 47391616 kB, free = 46536844 kB
+# node0 meminfo  : total = 47391616 kB, free = 46548348 kB
 # node0 cpu list : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# node1 meminfo  : total = 49539612 kB, free = 48908820 kB
-# node1 cpu list : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# node1 meminfo  : total = 49539612 kB, free = 48897176 kB
+# node1 cpu list : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
 # pmu mappings: intel_pt = 8, uncore_cha_1 = 25, uncore_irp_3 = 49, software = 1, uncore_imc_5 = 18, uncore_m3upi_0 = 21, uncore_iio_free_running_5 = 45, uncore_irp_1 = 47, uncore_m2m_1 = 12, uncore_imc_3 = 16, uncore_cha_8 = 32, uncore_iio_free_running_3 = 43, uncore_imc_1 = 14, uncore_upi_1 = 20, power = 10, uncore_cha_6 = 30, uncore_iio_free_running_1 = 41, uncore_iio_4 = 38, uprobe = 7, cpu = 4, uncore_cha_4 = 28, uncore_iio_2 = 36, cstate_core = 53, breakpoint = 5, uncore_cha_2 = 26, uncore_irp_4 = 50, uncore_m3upi_1 = 22, uncore_iio_0 = 34, tracepoint = 2, uncore_cha_0 = 24, uncore_irp_2 = 48, cstate_pkg = 54, uncore_imc_4 = 17, uncore_cha_9 = 33, uncore_iio_free_running_4 = 44, uncore_ubox = 23, uncore_irp_0 = 46, uncore_m2m_0 = 11, uncore_imc_2 = 15, kprobe = 6, uncore_cha_7 = 31, uncore_iio_free_running_2 = 42, uncore_iio_5 = 39, uncore_imc_0 = 13, uncore_upi_0 = 19, uncore_cha_5 = 29, uncore_iio_free_running_0 = 40, uncore_pcu = 52, msr = 9, uncore_iio_3 = 37, uncore_cha_3
  = 27, uncore_irp_5 = 51, uncore_iio_1 = 35
 # CPU cache info:
 #  L1 Data                 32K [0,20]
 #  L1 Instruction          32K [0,20]
+#  L1 Data                 32K [1,21]
+#  L1 Instruction          32K [1,21]
 #  L1 Data                 32K [2,22]
 #  L1 Instruction          32K [2,22]
 #  L1 Data                 32K [3,23]
@@ -123,9 +125,8 @@
 #  L1 Instruction          32K [18,38]
 #  L1 Data                 32K [19,39]
 #  L1 Instruction          32K [19,39]
-#  L1 Data                 32K [21]
-#  L1 Instruction          32K [21]
 #  L2 Unified            1024K [0,20]
+#  L2 Unified            1024K [1,21]
 #  L2 Unified            1024K [2,22]
 #  L2 Unified            1024K [3,23]
 #  L2 Unified            1024K [4,24]
@@ -144,12 +145,11 @@
 #  L2 Unified            1024K [17,37]
 #  L2 Unified            1024K [18,38]
 #  L2 Unified            1024K [19,39]
-#  L2 Unified            1024K [21]
 #  L3 Unified           14080K [0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38]
-#  L3 Unified           14080K [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39]
...
#3 lore raw

Re: [PATCH V5 1/1] perf: event preserve and create across cpu hotplug

Quoted 35 lines · Jiri Olsa
Quoted 26 lines · Mukesh Ojha

Perf framework doesn't allow preserving CPU events across CPU hotplugs. The events are scheduled out as and when the CPU walks offline. Moreover, the framework also doesn't allow the clients to create events on an offline CPU. As a result, the clients have to keep on monitoring the CPU state until it comes back online.

Therefore, introducing the perf framework to support creation and preserving of (CPU) events for offline CPUs. Through this, the CPU's online state would be transparent to the client and it not have to worry about monitoring the CPU's state. Success would be returned to the client even while creating the event on an offline CPU. If during the lifetime of the event the CPU walks offline, the event would be preserved and would continue to count as soon as (and if) the CPU comes back online.

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Quoted 4 lines

---
Change in V5:
=============
- Rebased it.

note that we might need to change how we store cpu topology, now that it can change during the sampling.. like below it's the comparison of header data with and without cpu 1

Quoted 5 lines

I think some of the report code checks on topology or caches and it might get confused

perhaps we could watch cpu topology in record and update the data as we see it changing.. future TODO list ;-)

Thanks Jiri for pointing it out.
I will take a look at perf record header update code.

It would be good, if we get more reviews on this patch as it is core event framework change.

-Mukesh

Quoted 75 lines

perf stat is probably fine

jirka

---

(quoted diff) +10 -8
-# nrcpus online : 39
+# nrcpus online : 40
  # nrcpus avail : 40
  # cpudesc : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
  # cpuid : GenuineIntel,6,85,4
...
  # sibling sockets : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling sockets : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling sockets : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
  # sibling dies    : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling dies    : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling dies    : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
  # sibling threads : 0,20
+# sibling threads : 1,21
  # sibling threads : 2,22
  # sibling threads : 3,23
  # sibling threads : 4,24
@@ -38,9 +39,8 @@
  # sibling threads : 17,37
  # sibling threads : 18,38
  # sibling threads : 19,39
-# sibling threads : 21
  # CPU 0: Core ID 0, Die ID 0, Socket ID 0
-# CPU 1: Core ID -1, Die ID -1, Socket ID -1
+# CPU 1: Core ID 0, Die ID 0, Socket ID 1
  # CPU 2: Core ID 4, Die ID 0, Socket ID 0
  # CPU 3: Core ID 4, Die ID 0, Socket ID 1
  # CPU 4: Core ID 1, Die ID 0, Socket ID 0
@@ -79,14 +79,16 @@
  # CPU 37: Core ID 9, Die ID 0, Socket ID 1
  # CPU 38: Core ID 10, Die ID 0, Socket ID 0
  # CPU 39: Core ID 10, Die ID 0, Socket ID 1
-# node0 meminfo  : total = 47391616 kB, free = 46536844 kB
+# node0 meminfo  : total = 47391616 kB, free = 46548348 kB
  # node0 cpu list : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# node1 meminfo  : total = 49539612 kB, free = 48908820 kB
-# node1 cpu list : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# node1 meminfo  : total = 49539612 kB, free = 48897176 kB
+# node1 cpu list : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
  # pmu mappings: intel_pt = 8, uncore_cha_1 = 25, uncore_irp_3 = 49, software = 1, uncore_imc_5 = 18, uncore_m3upi_0 = 21, uncore_iio_free_running_5 = 45, uncore_irp_1 = 47, uncore_m2m_1 = 12, uncore_imc_3 = 16, uncore_cha_8 = 32, uncore_iio_free_running_3 = 43, uncore_imc_1 = 14, uncore_upi_1 = 20, power = 10, uncore_cha_6 = 30, uncore_iio_free_running_1 = 41, uncore_iio_4 = 38, uprobe = 7, cpu = 4, uncore_cha_4 = 28, uncore_iio_2 = 36, cstate_core = 53, breakpoint = 5, uncore_cha_2 = 26, uncore_irp_4 = 50, uncore_m3upi_1 = 22, uncore_iio_0 = 34, tracepoint = 2, uncore_cha_0 = 24, uncore_irp_2 = 48, cstate_pkg = 54, uncore_imc_4 = 17, uncore_cha_9 = 33, uncore_iio_free_running_4 = 44, uncore_ubox = 23, uncore_irp_0 = 46, uncore_m2m_0 = 11, uncore_imc_2 = 15, kprobe = 6, uncore_cha_7 = 31, uncore_iio_free_running_2 = 42, uncore_iio_5 = 39, uncore_imc_0 = 13, uncore_upi_0 = 19, uncore_cha_5 = 29, uncore_iio_free_running_0 = 40, uncore_pcu = 52, msr = 9, uncore_iio_3 = 37, uncore_cha_3
   = 27, uncore_irp_5 = 51, uncore_iio_1 = 35
  # CPU cache info:
  #  L1 Data                 32K [0,20]
  #  L1 Instruction          32K [0,20]
+#  L1 Data                 32K [1,21]
+#  L1 Instruction          32K [1,21]
  #  L1 Data                 32K [2,22]
  #  L1 Instruction          32K [2,22]

# L1 Data 32K [3,23]

(quoted diff) +1 -4
@@ -123,9 +125,8 @@
  #  L1 Instruction          32K [18,38]
  #  L1 Data                 32K [19,39]
  #  L1 Instruction          32K [19,39]
-#  L1 Data                 32K [21]
-#  L1 Instruction          32K [21]
  #  L2 Unified            1024K [0,20]
+#  L2 Unified            1024K [1,21]
  #  L2 Unified            1024K [2,22]
  #  L2 Unified            1024K [3,23]
  #  L2 Unified            1024K [4,24]
@@ -144,12 +145,11 @@
  #  L2 Unified            1024K [17,37]
  #  L2 Unified            1024K [18,38]
  #  L2 Unified            1024K [19,39]
-#  L2 Unified            1024K [21]
  #  L3 Unified           14080K [0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38]
-#  L3 Unified           14080K [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39]
...
#4 lore raw

Re: [PATCH V5 1/1] perf: event preserve and create across cpu hotplug

In reply to Jiri Olsa
Quoted 35 lines · Jiri Olsa
Quoted 26 lines · Mukesh Ojha

Perf framework doesn't allow preserving CPU events across CPU hotplugs. The events are scheduled out as and when the CPU walks offline. Moreover, the framework also doesn't allow the clients to create events on an offline CPU. As a result, the clients have to keep on monitoring the CPU state until it comes back online.

Therefore, introducing the perf framework to support creation and preserving of (CPU) events for offline CPUs. Through this, the CPU's online state would be transparent to the client and it not have to worry about monitoring the CPU's state. Success would be returned to the client even while creating the event on an offline CPU. If during the lifetime of the event the CPU walks offline, the event would be preserved and would continue to count as soon as (and if) the CPU comes back online.

Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Quoted 4 lines

---
Change in V5:
=============
- Rebased it.

note that we might need to change how we store cpu topology, now that it can change during the sampling.. like below it's the comparison of header data with and without cpu 1

Quoted 5 lines

I think some of the report code checks on topology or caches and it might get confused

perhaps we could watch cpu topology in record and update the data as we see it changing.. future TODO list ;-)

Hi Jiri,

Can we do something like below  to address issue  related to header update while perf record with offline cpus.

tools/perf/builtin-record.c +0 -0
@@ -1432,7 +1432,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)

                opts->no_bpf_event = true;
        }

-       err = record__synthesize(rec, false);
+       err = record__synthesize(rec, true);
        if (err < 0)
                goto out_child;

@@ -1652,7 +1652,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv) } else
                status = err;

-       record__synthesize(rec, true);
+       record__synthesize(rec, false);
        /* this will be recalculated during process_buildids() */ rec->samples = 0;

Thanks.
Mukesh

Quoted 74 lines

perf stat is probably fine

jirka

---

(quoted diff) +11 -12
-# nrcpus online : 39
+# nrcpus online : 40
  # nrcpus avail : 40
  # cpudesc : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
  # cpuid : GenuineIntel,6,85,4
...
  # sibling sockets : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling sockets : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling sockets : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
  # sibling dies    : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling dies    : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling dies    : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
  # sibling threads : 0,20
+# sibling threads : 1,21
  # sibling threads : 2,22
  # sibling threads : 3,23
  # sibling threads : 4,24
@@ -38,9 +39,8 @@
  # sibling threads : 17,37
  # sibling threads : 18,38
  # sibling threads : 19,39
-# sibling threads : 21
  # CPU 0: Core ID 0, Die ID 0, Socket ID 0
-# CPU 1: Core ID -1, Die ID -1, Socket ID -1
+# CPU 1: Core ID 0, Die ID 0, Socket ID 1
  # CPU 2: Core ID 4, Die ID 0, Socket ID 0
  # CPU 3: Core ID 4, Die ID 0, Socket ID 1
  # CPU 4: Core ID 1, Die ID 0, Socket ID 0
@@ -79,14 +79,16 @@
  # CPU 37: Core ID 9, Die ID 0, Socket ID 1
  # CPU 38: Core ID 10, Die ID 0, Socket ID 0
  # CPU 39: Core ID 10, Die ID 0, Socket ID 1
-# node0 meminfo  : total = 47391616 kB, free = 46536844 kB
+# node0 meminfo  : total = 47391616 kB, free = 46548348 kB
  # node0 cpu list : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# node1 meminfo  : total = 49539612 kB, free = 48908820 kB
-# node1 cpu list : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# node1 meminfo  : total = 49539612 kB, free = 48897176 kB
+# node1 cpu list : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
  # pmu mappings: intel_pt = 8, uncore_cha_1 = 25, uncore_irp_3 = 49, software = 1, uncore_imc_5 = 18, uncore_m3upi_0 = 21, uncore_iio_free_running_5 = 45, uncore_irp_1 = 47, uncore_m2m_1 = 12, uncore_imc_3 = 16, uncore_cha_8 = 32, uncore_iio_free_running_3 = 43, uncore_imc_1 = 14, uncore_upi_1 = 20, power = 10, uncore_cha_6 = 30, uncore_iio_free_running_1 = 41, uncore_iio_4 = 38, uprobe = 7, cpu = 4, uncore_cha_4 = 28, uncore_iio_2 = 36, cstate_core = 53, breakpoint = 5, uncore_cha_2 = 26, uncore_irp_4 = 50, uncore_m3upi_1 = 22, uncore_iio_0 = 34, tracepoint = 2, uncore_cha_0 = 24, uncore_irp_2 = 48, cstate_pkg = 54, uncore_imc_4 = 17, uncore_cha_9 = 33, uncore_iio_free_running_4 = 44, uncore_ubox = 23, uncore_irp_0 = 46, uncore_m2m_0 = 11, uncore_imc_2 = 15, kprobe = 6, uncore_cha_7 = 31, uncore_iio_free_running_2 = 42, uncore_iio_5 = 39, uncore_imc_0 = 13, uncore_upi_0 = 19, uncore_cha_5 = 29, uncore_iio_free_running_0 = 40, uncore_pcu = 52, msr = 9, uncore_iio_3 = 37, uncore_cha_3 = 27, uncore_irp_5 = 51, uncore_iio_1 = 35
  # CPU cache info:
  #  L1 Data                 32K [0,20]
  #  L1 Instruction          32K [0,20]
+#  L1 Data                 32K [1,21]
+#  L1 Instruction          32K [1,21]
  #  L1 Data                 32K [2,22]
  #  L1 Instruction          32K [2,22]
  #  L1 Data                 32K [3,23]
@@ -123,9 +125,8 @@
  #  L1 Instruction          32K [18,38]
  #  L1 Data                 32K [19,39]
  #  L1 Instruction          32K [19,39]
-#  L1 Data                 32K [21]
-#  L1 Instruction          32K [21]
  #  L2 Unified            1024K [0,20]
+#  L2 Unified            1024K [1,21]
  #  L2 Unified            1024K [2,22]
  #  L2 Unified            1024K [3,23]
  #  L2 Unified            1024K [4,24]
@@ -144,12 +145,11 @@
  #  L2 Unified            1024K [17,37]
  #  L2 Unified            1024K [18,38]
  #  L2 Unified            1024K [19,39]
-#  L2 Unified            1024K [21]
  #  L3 Unified           14080K [0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38]
-#  L3 Unified           14080K [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39]
...