perf: Drop leaked kernel samples
cover perf: Drop leaked kernel samples
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project.
Now this patch set uses sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:
0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.
For rr it can write 1 to /sys/devices/cpu/perf_allow_sample_leakage to keep original system behavior.
Jin Yao (2):
perf/core: Use sysctl to turn on/off dropping leaked kernel samples perf Documentation: Introduce the sysctl perf_allow_sample_leakage
kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++ tools/perf/Documentation/perf-record.txt | 14 ++++++++ 2 files changed, 72 insertions(+)
Signature · 1 line
2.7.4
PATCH 1/2 perf/core: Use sysctl to turn on/off dropping leaked kernel samples
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '6a8a75f32357 ("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.
Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:
0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.
For rr it can write 1 to /sys/devices/cpu/perf_allow_sample_leakage.
For example,
root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 0 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio
........ ....... ............. ................
47.01% div div [.] main
20.74% div libc-2.23.so [.] __random_r
15.59% div libc-2.23.so [.] __random
8.68% div div [.] compute_flag
4.48% div libc-2.23.so [.] rand
3.50% div div [.] rand@plt
0.00% div ld-2.23.so [.] do_lookup_x
0.00% div ld-2.23.so [.] memcmp
0.00% div ld-2.23.so [.] _dl_start
0.00% div ld-2.23.so [.] _startThere is no kernel symbol reported.
root@skl:/tmp# echo 1 > /sys/devices/cpu/perf_allow_sample_leakage root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 1 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio
........ ....... ................ .............
47.53% div div [.] main
20.62% div libc-2.23.so [.] __random_r
15.32% div libc-2.23.so [.] __random
8.66% div div [.] compute_flag
4.53% div libc-2.23.so [.] rand
3.34% div div [.] rand@plt
0.00% div [kernel.vmlinux] [k] apic_timer_interrupt
0.00% div libc-2.23.so [.] intel_check_word
0.00% div ld-2.23.so [.] brk
0.00% div [kernel.vmlinux] [k] page_fault
0.00% div ld-2.23.so [.] _startWe can see the kernel symbols apic_timer_interrupt and page_fault.
--- kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
kernel/events/core.c +58 -0
@@ -7721,6 +7721,28 @@ int perf_event_account_interrupt(struct perf_event *event)return __perf_event_account_interrupt (event,1 ); } +static int perf_allow_sample_leakage__read_mostly ; + +static bool sample_is_allowed (struct perf_event *event,struct pt_regs *regs) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +if (allow_leakage) +return true; + +/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; + +return true; +} +/* * Generic event overflow handling, sampling. */
@@ -7742,6 +7764,12 @@ static int __perf_event_overflow(struct perf_event *event, ret =__perf_event_account_interrupt (event, throttle);/* +* For security, drop the skid kernel samples if necessary. +*/ +if (!sample_is_allowed (event, regs)) +return ret; + +/* * XXX event_limit might not quite work as expected on inherited * events */
@@ -9500,9 +9528,39 @@ perf_event_mux_interval_ms_store(struct device *dev, }static DEVICE_ATTR_RW (perf_event_mux_interval_ms); +static ssize_t +perf_allow_sample_leakage_show (struct device *dev, +struct device_attribute *attr,char *page) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +return snprintf (page,PAGE_SIZE -1 ,"%d\n" , allow_leakage); +} + +static ssize_t +perf_allow_sample_leakage_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +int allow_leakage, ret; + + ret =kstrtoint (buf,0 , &allow_leakage); +if (ret) +return ret; + +if (allow_leakage !=0 && allow_leakage !=1 ) +return -EINVAL ; + +WRITE_ONCE (perf_allow_sample_leakage, allow_leakage); + +return count; +} +static DEVICE_ATTR_RW (perf_allow_sample_leakage); +static struct attribute *pmu_dev_attrs[] = { &dev_attr_type.attr, &dev_attr_perf_event_mux_interval_ms.attr, + &dev_attr_perf_allow_sample_leakage.attr,NULL , };ATTRIBUTE_GROUPS (pmu_dev);
Signature · 1 line
2.7.4
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 72 lines · Jin Yao
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '
6a8a75f32357("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.For rr it can write 1 to
/sys/devices/cpu/perf_allow_sample_leakage.For example,
root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 0 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ............. ................
47.01% div div [.] main 20.74% div libc-2.23.so [.] __random_r 15.59% div libc-2.23.so [.] __random 8.68% div div [.] compute_flag 4.48% div libc-2.23.so [.] rand 3.50% div div [.] rand@plt 0.00% div ld-2.23.so [.] do_lookup_x 0.00% div ld-2.23.so [.] memcmp 0.00% div ld-2.23.so [.] _dl_start 0.00% div ld-2.23.so [.] _startThere is no kernel symbol reported.
root@skl:/tmp# echo 1 > /sys/devices/cpu/perf_allow_sample_leakage root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 1 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ................ .............
47.53% div div [.] main 20.62% div libc-2.23.so [.] __random_r 15.32% div libc-2.23.so [.] __random 8.66% div div [.] compute_flag 4.53% div libc-2.23.so [.] rand 3.34% div div [.] rand@plt 0.00% div [kernel.vmlinux] [k] apic_timer_interrupt 0.00% div libc-2.23.so [.] intel_check_word 0.00% div ld-2.23.so [.] brk 0.00% div [kernel.vmlinux] [k] page_fault 0.00% div ld-2.23.so [.] _start
Quoted 2 lines
We can see the kernel symbols
apic_timer_interruptandpage_fault.
These kernel symbols do not match your description here. How much skid do you think you have here?
You're saying you are at the user level, you get a counter overflow, and the interrupted IP lands in the kernel
because you where there by the time the interrupt is delivered. How many instructions does it take to get
from user land to apic_timer_interrupt() or page_fault()? These functions are not right at the kernel entry,
I believe. So how did you get there, the skid must have been VERY big or symbolization has a problem.
Quoted 94 lines
Signed-off-by Jin Yao <[email protected]>--- kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)kernel/events/core.c +58
-1@@ -7721,6 +7721,28 @@ int perf_event_account_interrupt(struct perf_event *event)return __perf_event_account_interrupt (event,1 ); } +static int perf_allow_sample_leakage__read_mostly ; + +static bool sample_is_allowed (struct perf_event *event,struct pt_regs *regs) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +if (allow_leakage) +return true; + +/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; + +return true; +} +/* * Generic event overflow handling, sampling. */ @@ -7742,6 +7764,12 @@ static int __perf_event_overflow(struct perf_event *event, ret =__perf_event_account_interrupt (event, throttle);/* +* For security, drop the skid kernel samples if necessary. +*/ +if (!sample_is_allowed (event, regs)) +return ret; + +/* * XXX event_limit might not quite work as expected on inherited * events */ @@ -9500,9 +9528,39 @@ perf_event_mux_interval_ms_store(struct device *dev, }static DEVICE_ATTR_RW (perf_event_mux_interval_ms); +static ssize_t +perf_allow_sample_leakage_show (struct device *dev, +struct device_attribute *attr,char *page) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +return snprintf (page,PAGE_SIZE -1 ,"%d\n" , allow_leakage); +} + +static ssize_t +perf_allow_sample_leakage_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +int allow_leakage, ret; + + ret =kstrtoint (buf,0 , &allow_leakage); +if (ret) +return ret; + +if (allow_leakage !=0 && allow_leakage !=1 ) +return -EINVAL ; + +WRITE_ONCE (perf_allow_sample_leakage, allow_leakage); + +return count; +} +static DEVICE_ATTR_RW (perf_allow_sample_leakage); +static struct attribute *pmu_dev_attrs[] = { &dev_attr_type.attr, &dev_attr_perf_event_mux_interval_ms.attr, + &dev_attr_perf_allow_sample_leakage.attr,NULL , };ATTRIBUTE_GROUPS (pmu_dev); --2.7.4
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 75 lines · Stephane Eranian
Quoted 72 lines · Jin Yao
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '
6a8a75f32357("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.For rr it can write 1 to
/sys/devices/cpu/perf_allow_sample_leakage.For example,
root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 0 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ............. ................
47.01% div div [.] main 20.74% div libc-2.23.so [.] __random_r 15.59% div libc-2.23.so [.] __random 8.68% div div [.] compute_flag 4.48% div libc-2.23.so [.] rand 3.50% div div [.] rand@plt 0.00% div ld-2.23.so [.] do_lookup_x 0.00% div ld-2.23.so [.] memcmp 0.00% div ld-2.23.so [.] _dl_start 0.00% div ld-2.23.so [.] _startThere is no kernel symbol reported.
root@skl:/tmp# echo 1 > /sys/devices/cpu/perf_allow_sample_leakage root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 1 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ................ .............
47.53% div div [.] main 20.62% div libc-2.23.so [.] __random_r 15.32% div libc-2.23.so [.] __random 8.66% div div [.] compute_flag 4.53% div libc-2.23.so [.] rand 3.34% div div [.] rand@plt 0.00% div [kernel.vmlinux] [k] apic_timer_interrupt 0.00% div libc-2.23.so [.] intel_check_word 0.00% div ld-2.23.so [.] brk 0.00% div [kernel.vmlinux] [k] page_fault 0.00% div ld-2.23.so [.] _startQuoted 2 lines
We can see the kernel symbols
apic_timer_interruptandpage_fault.
Quoted 11 lines
These kernel symbols do not match your description here. How much skid do you think you have here?
You're saying you are at the user level, you get a counter overflow, and the interrupted IP lands in the kernel
because you where there by the time the interrupt is delivered. How many instructions does it take to get
from user land toapic_timer_interrupt()orpage_fault()? These functions are not right at the kernel entry,
I believe. So how did you get there, the skid must have been VERY big or symbolization has a problem.
I'm testing with the latest perf/core branch (4.17+). Again I test with Linux's vmstat (not test with my application).
perf record -e cycles:u vmstat 1
perf script -F ip
7f84e2b0bc30
7f84e2b0bc30
7f84e2b0bc30
7f84e2b0bc30
ffffffffb7a01070
7f84e2b243f0
7f84e2b11891
7f84e2b27f5e
7f84e25a3b26
7f84e25680f5cat /proc/kallsyms | grep page_fault
....ffffffffb7a01070 T page_faultffffffffb7a010a0 T async_page_fault
....
So one sample (ip ffffffffb7a01070) hits on page_fault.
Maybe you can have a try too. :)
Thanks
Jin Yao
Quoted 94 lines
Quoted 94 lines
Signed-off-by Jin Yao <[email protected]>--- kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)kernel/events/core.c +58
-1@@ -7721,6 +7721,28 @@ int perf_event_account_interrupt(struct perf_event *event)return __perf_event_account_interrupt (event,1 ); } +static int perf_allow_sample_leakage__read_mostly ; + +static bool sample_is_allowed (struct perf_event *event,struct pt_regs *regs) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +if (allow_leakage) +return true; + +/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; + +return true; +} +/* * Generic event overflow handling, sampling. */ @@ -7742,6 +7764,12 @@ static int __perf_event_overflow(struct perf_event *event, ret =__perf_event_account_interrupt (event, throttle);/* +* For security, drop the skid kernel samples if necessary. +*/ +if (!sample_is_allowed (event, regs)) +return ret; + +/* * XXX event_limit might not quite work as expected on inherited * events */ @@ -9500,9 +9528,39 @@ perf_event_mux_interval_ms_store(struct device *dev, }static DEVICE_ATTR_RW (perf_event_mux_interval_ms); +static ssize_t +perf_allow_sample_leakage_show (struct device *dev, +struct device_attribute *attr,char *page) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +return snprintf (page,PAGE_SIZE -1 ,"%d\n" , allow_leakage); +} + +static ssize_t +perf_allow_sample_leakage_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +int allow_leakage, ret; + + ret =kstrtoint (buf,0 , &allow_leakage); +if (ret) +return ret; + +if (allow_leakage !=0 && allow_leakage !=1 ) +return -EINVAL ; + +WRITE_ONCE (perf_allow_sample_leakage, allow_leakage); + +return count; +} +static DEVICE_ATTR_RW (perf_allow_sample_leakage); +static struct attribute *pmu_dev_attrs[] = { &dev_attr_type.attr, &dev_attr_perf_event_mux_interval_ms.attr, + &dev_attr_perf_allow_sample_leakage.attr,NULL , };ATTRIBUTE_GROUPS (pmu_dev); --2.7.4
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 114 lines · Yao
Quoted 75 lines · Stephane Eranian
Quoted 72 lines · Jin Yao
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '
6a8a75f32357("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.For rr it can write 1 to
/sys/devices/cpu/perf_allow_sample_leakage.For example,
root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 0 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ............. ................
47.01% div div [.] main 20.74% div libc-2.23.so [.] __random_r 15.59% div libc-2.23.so [.] __random 8.68% div div [.] compute_flag 4.48% div libc-2.23.so [.] rand 3.50% div div [.] rand@plt 0.00% div ld-2.23.so [.] do_lookup_x 0.00% div ld-2.23.so [.] memcmp 0.00% div ld-2.23.so [.] _dl_start 0.00% div ld-2.23.so [.] _startThere is no kernel symbol reported.
root@skl:/tmp# echo 1 > /sys/devices/cpu/perf_allow_sample_leakage root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 1 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ................ .............
47.53% div div [.] main 20.62% div libc-2.23.so [.] __random_r 15.32% div libc-2.23.so [.] __random 8.66% div div [.] compute_flag 4.53% div libc-2.23.so [.] rand 3.34% div div [.] rand@plt 0.00% div [kernel.vmlinux] [k] apic_timer_interrupt 0.00% div libc-2.23.so [.] intel_check_word 0.00% div ld-2.23.so [.] brk 0.00% div [kernel.vmlinux] [k] page_fault 0.00% div ld-2.23.so [.] _startQuoted 2 lines
We can see the kernel symbols
apic_timer_interruptandpage_fault.Quoted 11 lines
These kernel symbols do not match your description here. How much skid do you think you have here?
You're saying you are at the user level, you get a counter overflow, and the interrupted IP lands in the kernel
because you where there by the time the interrupt is delivered. How many instructions does it take to get
from user land toapic_timer_interrupt()orpage_fault()? These functions are not right at the kernel entry,
I believe. So how did you get there, the skid must have been VERY big or symbolization has a problem.I'm testing with the latest perf/core branch (4.17+). Again I test with Linux's vmstat (not test with my application).
perf record -e cycles:u vmstat 1
perf script -F ip7f84e2b0bc30 7f84e2b0bc30 7f84e2b0bc30 7f84e2b0bc30 ffffffffb7a01070 7f84e2b243f0 7f84e2b11891 7f84e2b27f5e 7f84e25a3b26 7f84e25680f5cat
/proc/kallsyms| greppage_fault
....ffffffffb7a01070Tpage_faultffffffffb7a010a0Tasync_page_fault
....
Quoted 4 lines
So one sample (ip
ffffffffb7a01070) hits onpage_fault.Maybe you can have a try too. :)
Ok, I tried and checked! These symbols are all in entry_64.S, these are the first instructions executed on the timer intr or fault. So it looks normal that they show up due to the interrupt skid, even if the skid is 1 cycle.
PEBS, especially when using precise=1 could also show these symbols.
Quoted 98 lines
Thanks
Jin YaoQuoted 94 lines
Quoted 94 lines
Signed-off-by Jin Yao <[email protected]>--- kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)kernel/events/core.c +58
-1@@ -7721,6 +7721,28 @@ int perf_event_account_interrupt(struct perf_event *event)return __perf_event_account_interrupt (event,1 ); } +static int perf_allow_sample_leakage__read_mostly ; + +static bool sample_is_allowed (struct perf_event *event,struct pt_regs *regs) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +if (allow_leakage) +return true; + +/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; + +return true; +} +/* * Generic event overflow handling, sampling. */ @@ -7742,6 +7764,12 @@ static int __perf_event_overflow(struct perf_event *event, ret =__perf_event_account_interrupt (event, throttle);/* +* For security, drop the skid kernel samples if necessary. +*/ +if (!sample_is_allowed (event, regs)) +return ret; + +/* * XXX event_limit might not quite work as expected on inherited * events */ @@ -9500,9 +9528,39 @@ perf_event_mux_interval_ms_store(struct device *dev, }static DEVICE_ATTR_RW (perf_event_mux_interval_ms); +static ssize_t +perf_allow_sample_leakage_show (struct device *dev, +struct device_attribute *attr,char *page) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +return snprintf (page,PAGE_SIZE -1 ,"%d\n" , allow_leakage); +} + +static ssize_t +perf_allow_sample_leakage_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +int allow_leakage, ret; + + ret =kstrtoint (buf,0 , &allow_leakage); +if (ret) +return ret; + +if (allow_leakage !=0 && allow_leakage !=1 ) +return -EINVAL ; + +WRITE_ONCE (perf_allow_sample_leakage, allow_leakage); + +return count; +} +static DEVICE_ATTR_RW (perf_allow_sample_leakage); +static struct attribute *pmu_dev_attrs[] = { &dev_attr_type.attr, &dev_attr_perf_event_mux_interval_ms.attr, + &dev_attr_perf_allow_sample_leakage.attr,NULL , };ATTRIBUTE_GROUPS (pmu_dev); --2.7.4
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
In reply to Jin YaoQuoted 96 lines · Jin Yao
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '
6a8a75f32357("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.For rr it can write 1 to
/sys/devices/cpu/perf_allow_sample_leakage.For example,
root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 0 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ............. ................
47.01% div div [.] main 20.74% div libc-2.23.so [.] __random_r 15.59% div libc-2.23.so [.] __random 8.68% div div [.] compute_flag 4.48% div libc-2.23.so [.] rand 3.50% div div [.] rand@plt 0.00% div ld-2.23.so [.] do_lookup_x 0.00% div ld-2.23.so [.] memcmp 0.00% div ld-2.23.so [.] _dl_start 0.00% div ld-2.23.so [.] _startThere is no kernel symbol reported.
root@skl:/tmp# echo 1 > /sys/devices/cpu/perf_allow_sample_leakage root@skl:/tmp# cat /sys/devices/cpu/perf_allow_sample_leakage 1 root@skl:/tmp# perf record -e cycles:u ./div root@skl:/tmp# perf report --stdio........ ....... ................ .............
47.53% div div [.] main 20.62% div libc-2.23.so [.] __random_r 15.32% div libc-2.23.so [.] __random 8.66% div div [.] compute_flag 4.53% div libc-2.23.so [.] rand 3.34% div div [.] rand@plt 0.00% div [kernel.vmlinux] [k] apic_timer_interrupt 0.00% div libc-2.23.so [.] intel_check_word 0.00% div ld-2.23.so [.] brk 0.00% div [kernel.vmlinux] [k] page_fault 0.00% div ld-2.23.so [.] _startWe can see the kernel symbols
apic_timer_interruptandpage_fault.Signed-off-by Jin Yao <[email protected]>--- kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)kernel/events/core.c +9
-0@@ -7721,6 +7721,28 @@ int perf_event_account_interrupt(struct perf_event *event)return __perf_event_account_interrupt (event,1 ); } +static int perf_allow_sample_leakage__read_mostly ; + +static bool sample_is_allowed (struct perf_event *event,struct pt_regs *regs) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +if (allow_leakage) +return true; +
kernel/events/core.c int perf_event_account_interrupt(struct perf_event *event) · line 7742
Quoted 10 lines
kernel/events/core.c +10
-0+/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; +
And how does that filter PEBS or LBR records?
Quoted 62 lines
kernel/events/core.c +39
-1+return true; +} +/* * Generic event overflow handling, sampling. */ @@ -7742,6 +7764,12 @@ static int __perf_event_overflow(struct perf_event *event, ret =__perf_event_account_interrupt (event, throttle);/* +* For security, drop the skid kernel samples if necessary. +*/ +if (!sample_is_allowed (event, regs)) +return ret; + +/* * XXX event_limit might not quite work as expected on inherited * events */ @@ -9500,9 +9528,39 @@ perf_event_mux_interval_ms_store(struct device *dev, }static DEVICE_ATTR_RW (perf_event_mux_interval_ms); +static ssize_t +perf_allow_sample_leakage_show (struct device *dev, +struct device_attribute *attr,char *page) +{ +int allow_leakage =READ_ONCE (perf_allow_sample_leakage); + +return snprintf (page,PAGE_SIZE -1 ,"%d\n" , allow_leakage); +} + +static ssize_t +perf_allow_sample_leakage_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +int allow_leakage, ret; + + ret =kstrtoint (buf,0 , &allow_leakage); +if (ret) +return ret; + +if (allow_leakage !=0 && allow_leakage !=1 ) +return -EINVAL ; + +WRITE_ONCE (perf_allow_sample_leakage, allow_leakage); + +return count; +} +static DEVICE_ATTR_RW (perf_allow_sample_leakage); +static struct attribute *pmu_dev_attrs[] = { &dev_attr_type.attr, &dev_attr_perf_event_mux_interval_ms.attr, + &dev_attr_perf_allow_sample_leakage.attr,NULL , };ATTRIBUTE_GROUPS (pmu_dev); --2.7.4
Peter Zijlstra
maintainerRe: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 11 lines · Stephane Eranian
kernel/events/core.c int perf_event_account_interrupt(struct perf_event *event) · line 7742
Quoted 10 lines · Jin Yao
kernel/events/core.c +10
-0+/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; +
Quoted 1 line
And how does that filter PEBS or LBR records?
I suspect the user_mode() thing actually covers PEBS, but yes LBR might need additional filtering.
RE: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 12 lines · Stephane Eranian
Quoted 11 lines · Jin Yao
Quoted 10 lines
kernel/events/core.c +10
-0+/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; +
Quoted 4 lines
Quoted 1 line
And how does that filter PEBS or LBR records?
I suspect the
user_mode()thing actually covers PEBS, but yes LBR might need additional filtering.
I think the large PEBS still need to be specially handled.
Thanks,
Kan
Peter Zijlstra
maintainerRe: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 12 lines · Kan
Quoted 12 lines · Stephane Eranian
Quoted 11 lines · Jin Yao
Quoted 10 lines
kernel/events/core.c +10
-0+/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return false; +
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
And how does that filter PEBS or LBR records?
I suspect the
user_mode()thing actually covers PEBS, but yes LBR might need additional filtering.I think the large PEBS still need to be specially handled.
large pebs should be fine too, all pebs stuff eventually calls perf_event_output() with a synthesized register set.
Mark Rutland
reviewerRe: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
In reply to Jin YaoQuoted 19 lines · Jin Yao
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '
6a8a75f32357("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.
Quoted 7 lines
Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.
Does this need to be conditional at all?
At least for sampling the GPRs, we could do something like below unconditionally, which seems sufficient for my test cases.
Mark.
---->8----
kernel/events/core.c +20 -0
@@ -6359,6 +6359,24 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return callchain ?: &__empty_callchain; } +static struct pt_regs *perf_get_sample_regs (struct perf_event *event,struct pt_regs *regs) +{ +/* +* Due to interrupt latency (AKA "skid"), we may enter the kernel +* before taking an overflow, even if the PMU is only counting user +* events. +* +* If we're not counting kernel events, always use the user regs when +* sampling. +* +* TODO: how does this interact with guest sampling? +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return task_pt_regs (current); + +return regs; +} + void perf_prepare_sample(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event,
@@ -6366,6 +6384,8 @@ void perf_prepare_sample(struct perf_event_header *header, {u64 sample_type = event->attr.sample_type; + regs =perf_get_sample_regs (event, regs); + header->type =PERF_RECORD_SAMPLE ; header->size =sizeof (*header) + event->header_size;
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 3 lines · Mark Rutland
At least for sampling the GPRs, we could do something like below unconditionally, which seems sufficient for my test cases.
Ack.
The PEBS case may need checking, but maybe PEBS doesn't even have this issue?
Linus
Peter Zijlstra
maintainerRe: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Mark Rutland
At least for sampling the GPRs, we could do something like below unconditionally, which seems sufficient for my test cases.
Ack.
The PEBS case may need checking, but maybe PEBS doesn't even have this issue?
Sadly PEBS is susceptible as well, but the proposed patch actually works for that as well.
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
In reply to Mark RutlandQuoted 67 lines · Mark Rutland
Quoted 19 lines · Jin Yao
When doing sampling, for example:
perf record -e cycles:u ...
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project, which used sampling events to receive a signal on overflow. These signals were critical to the correct operation of rr.
See '
6a8a75f32357("Revert "perf/core: Drop kernel samples even though :u is specified"")' for detail.Quoted 7 lines
Now the idea is to use sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.Does this need to be conditional at all?
At least for sampling the GPRs, we could do something like below unconditionally, which seems sufficient for my test cases.
Mark.
---->8----
kernel/events/core.c +18
-0@@ -6359,6 +6359,24 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return callchain ?: &__empty_callchain; } +static struct pt_regs *perf_get_sample_regs (struct perf_event *event,struct pt_regs *regs) +{ +/* +* Due to interrupt latency (AKA "skid"), we may enter the kernel +* before taking an overflow, even if the PMU is only counting user +* events. +* +* If we're not counting kernel events, always use the user regs when +* sampling. +* +* TODO: how does this interact with guest sampling? +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return task_pt_regs (current); + +return regs; +} + void perf_prepare_sample(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event,
kernel/events/core.c void perf_prepare_sample(struct perf_event_header *header, · line 6391
Quoted 8 lines
kernel/events/core.c +2
-0@@ -6366,6 +6384,8 @@ void perf_prepare_sample(struct perf_event_header *header, {u64 sample_type = event->attr.sample_type; + regs =perf_get_sample_regs (event, regs); + header->type =PERF_RECORD_SAMPLE ; header->size =sizeof (*header) + event->header_size;
Hi Mark,
Thanks for providing the patch. I understand this approach.
In my opinion, the skid window is from counter overflow to interrupt delivered. While if the skid window is too *big* (e.g. user -> kernel), it should be not very useful. So personally, I'd prefer to drop the samples.
Thanks
Jin Yao
Peter Zijlstra
maintainerRe: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 5 lines · Yao
Thanks for providing the patch. I understand this approach.
In my opinion, the skid window is from counter overflow to interrupt delivered. While if the skid window is too *big* (e.g. user -> kernel), it should be not very useful. So personally, I'd prefer to drop the samples.
I really don't get your insitence on dropping the sample. Dropping samples is bad. Furthermore, doing what Mark suggests actually improves the result by reducing the skid, if the event happened before we entered (as it damn well should) then the user regs, which point at the entry site, are a better approximation than our in-kernel set.
So not only do you not loose the sample, you actually get a better sample.
Re: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Yao
Thanks for providing the patch. I understand this approach.
In my opinion, the skid window is from counter overflow to interrupt delivered. While if the skid window is too *big* (e.g. user -> kernel), it should be not very useful. So personally, I'd prefer to drop the samples.
Quoted 9 lines
I really don't get your insitence on dropping the sample. Dropping samples is bad. Furthermore, doing what Mark suggests actually improves the result by reducing the skid, if the event happened before we entered (as it damn well should) then the user regs, which point at the entry site, are a better approximation than our in-kernel set.
So not only do you not loose the sample, you actually get a better sample.
OK, that's fine, thanks!
I guess Mark will post this patch, right?
Anyway looks we don't need following patch (0-stuffing sample->ip to indicate perf tool that it is a leak sample), right?
kernel/events/core.c +18 -0
@@ -6361,6 +6361,21 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return callchain ?: &__empty_callchain; } +static bool sample_is_leaked (struct perf_event *event,struct pt_regs *regs) +{ +/* +* Due to interrupt latency (AKA "skid"), we may enter the +* kernel before taking an overflow, even if the PMU is only +* counting user events. +* To avoid leaking information to userspace, we must always +* reject kernel samples when exclude_kernel is set. +*/ +if (event->attr.exclude_kernel && !user_mode (regs)) +return true; + +return false; +} + void perf_prepare_sample(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event,
@@ -6480,6 +6495,9 @@ void perf_prepare_sample(struct perf_event_header *header,if (sample_type &PERF_SAMPLE_PHYS_ADDR ) data->phys_addr =perf_virt_to_phys (data->addr); + +if (sample_is_leaked (event, regs)) + data->ip =0 ; }static void __always_inline
tools/perf/util/event.h +1 -0
@@ -404,6 +404,7 @@ struct events_stats {u64 total_aux_lost;u64 total_aux_partial;u64 total_invalid_chains; +u64 total_dropped_samples;u32 nr_events[PERF_RECORD_HEADER_MAX ];u32 nr_non_filtered_samples;u32 nr_lost_warned;
tools/perf/util/session.c +6 -0
@@ -1269,6 +1269,12 @@ static int machines__deliver_event(struct machines *machines, ++evlist->stats.nr_unprocessable_samples;return 0 ; } + +if (sample->ip ==0 ) { +/* Drop the leaked kernel samples */ + ++evlist->stats.total_dropped_samples; +return 0 ; + } return perf_evlist__deliver_sample(evlist, tool, event,
sample, evsel, machine);
case PERF_RECORD_MMAP:
return tool->mmap(tool, event, sample, machine);Thanks
Jin Yao
Mark Rutland
reviewerRe: [PATCH v1 1/2] perf/core: Use sysctl to turn on/off dropping leaked kernel samples
Quoted 16 lines · Yao
Quoted 13 lines · Peter Zijlstra
Quoted 5 lines · Yao
Thanks for providing the patch. I understand this approach.
In my opinion, the skid window is from counter overflow to interrupt delivered. While if the skid window is too *big* (e.g. user -> kernel), it should be not very useful. So personally, I'd prefer to drop the samples.
I really don't get your insitence on dropping the sample. Dropping samples is bad. Furthermore, doing what Mark suggests actually improves the result by reducing the skid, if the event happened before we entered (as it damn well should) then the user regs, which point at the entry site, are a better approximation than our in-kernel set.
Quoted 7 lines
Quoted 3 lines
So not only do you not loose the sample, you actually get a better sample.
OK, that's fine, thanks!
I guess Mark will post this patch, right?
I'll try to spin something shortly -- I'm just figuring out how this should work with guest sampling.
Thanks,
Mark.
PATCH 2/2 perf Documentation: Introduce the sysctl perf_allow_sample_leakage
In reply to Jin YaoIntroduce a new sysctl /sys/devices/cpu/perf_allow_sample_leakage, which turns on/off dropping leaked kernel samples.
--- tools/perf/Documentation/perf-record.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/perf/Documentation/perf-record.txt +14 -0
@@ -93,6 +93,20 @@ OPTIONS prevent the shell interpretation. You also need to use --group on "perf report" to view group events together. + Note that if workload does a lot of kernel entry/exit we may see + kernel samples even if :u is specified. That is due to skid existing. + This might be a security issue because it can leak kernel address even + though kernel sampling support is disabled. We have a sysctl to turn + on/off the dropping of leaked kernel samples. + + /sys/devices/cpu/perf_allow_sample_leakage + + 0 - drop the leaked kernel samples, default option. + 1 - don't drop the leaked kernel samples. + + For example, write 1 to perf_allow_sample_leakage + echo 1 > /sys/devices/cpu/perf_allow_sample_leakage + --filter=<filter>:: Event filter. This option should follow a event selector (-e) which selects either tracepoint event(s) or a hardware trace PMU
Signature · 1 line
2.7.4
I strongly object to this patch as written. As I said when I originally reported[0] the regression introduced by the previous version of this patch a year ago.
"It seems like this change should, at a bare minimum, be limited to counters that actually perform sampling of register state when the interrupt fires. In our case, with the retired conditional branches counter restricted to counting userspace events only, it makes no difference that the PMU interrupt happened to be delivered in the kernel."
This means identifying which values of perf_event_attr::sample_type are security concerns (presumably PERF_SAMPLE_IP is, and PERF_SAMPLE_TIME is not, and someone needs to go through and decide on all of them) and filtering on those values for this new behavior.
And because rr sets its sample_type to 0, once you do that, the sysctl will not be necessary.
- Kyle
Quoted 25 lines · Jin Yao
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project.
Now this patch set uses sysctl to control the dropping of leaked kernel samples.
/sys/devices/cpu/perf_allow_sample_leakage:0 - default, drop the leaked kernel samples.
1 - don't drop the leaked kernel samples.For rr it can write 1 to
/sys/devices/cpu/perf_allow_sample_leakageto keep original system behavior.Jin Yao (2):
perf/core: Use sysctl to turn on/off dropping leaked kernel samples perf Documentation: Introduce the sysctlperf_allow_sample_leakage
Quoted 7 lines
kernel/events/core.c | 58 ++++++++++++++++++++++++++++++++ tools/perf/Documentation/perf-record.txt | 14 ++++++++ 2 files changed, 72 insertions(+)--
2.7.4
Quoted 16 lines · Kyle Huey
I strongly object to this patch as written. As I said when I originally reported[0] the regression introduced by the previous version of this patch a year ago.
"It seems like this change should, at a bare minimum, be limited to counters that actually perform sampling of register state when the interrupt fires. In our case, with the retired conditional branches counter restricted to counting userspace events only, it makes no difference that the PMU interrupt happened to be delivered in the kernel."
This means identifying which values of
perf_event_attr::sample_typeare security concerns (presumablyPERF_SAMPLE_IPis, andPERF_SAMPLE_TIMEis not, and someone needs to go through and decide on all of them) and filtering on those values for this new behavior.
Quoted 5 lines
And because rr sets its
sample_typeto 0, once you do that, the sysctl will not be necessary.- Kyle
Since rr sets sample_type to 0, the easiest way is to add checking like:
if (event->attr.sample_type) {
if (event->attr.exclude_kernel && !user_mode(regs))
return false;
}So the rr doesn't need to be changed and for other use cases the leaked kernel samples will be dropped.
But I don't like this is because:
1. It's too specific for rr case.
2. If we create a new sample_type, e.g. PERF_SAMPLE_ALLOW_LEAKAGE, the code will be:
if !(event->attr.sample_type & PERF_SAMPLE_ALLOW_LEAKAGE) {
if (event->attr.exclude_kernel && !user_mode(regs))
return false;
}But rr needs to add PERF_SAMPLE_ALLOW_LEAKAGE to sample_type since by default the bit is not set.
3. Sysctl is a more flexible way. It provides us with an option when we want to see if skid is existing, we can use sysctl to turn on that.
Thanks
Jin Yao
Quoted 33 lines · Yao
Quoted 17 lines · Kyle Huey
I strongly object to this patch as written. As I said when I originally reported[0] the regression introduced by the previous version of this patch a year ago.
"It seems like this change should, at a bare minimum, be limited to counters that actually perform sampling of register state when the interrupt fires. In our case, with the retired conditional branches counter restricted to counting userspace events only, it makes no difference that the PMU interrupt happened to be delivered in the kernel."
This means identifying which values of
perf_event_attr::sample_typeare security concerns (presumablyPERF_SAMPLE_IPis, andPERF_SAMPLE_TIMEis not, and someone needs to go through and decide on all of them) and filtering on those values for this new behavior.Quoted 5 lines
And because rr sets its
sample_typeto 0, once you do that, the sysctl will not be necessary.- Kyle
Since rr sets
sample_typeto 0, the easiest way is to add checking like:if (event->attr.sample_type) { if (event->attr.exclude_kernel && !user_mode(regs)) return false; }
Quoted 6 lines
So the rr doesn't need to be changed and for other use cases the leaked kernel samples will be dropped.
But I don't like this is because:
1. It's too specific for rr case.
Keeping existing software working is the first rule of kernel development!
There is no disclosure of kernel space state in the way rr uses this API, so there is no reason that this API should not keep working.
Quoted 10 lines
2. If we create a new
sample_type, e.g.PERF_SAMPLE_ALLOW_LEAKAGE, the code will be:if !(event->attr.sample_type & PERF_SAMPLE_ALLOW_LEAKAGE) { if (event->attr.exclude_kernel && !user_mode(regs)) return false; }But rr needs to add
PERF_SAMPLE_ALLOW_LEAKAGEtosample_typesince by default the bit is not set.
There's no reason to add a new PERF_SAMPLE flag. You need to audit the *existing* PERF_SAMPLE flags and figure out which ones are problems, and then do
if (event->attr.exclude_kernel && !user_mode(regs) &&
sampling_discloses_kernel_information(event->attr.sample_type)) {
return false;
}Quoted 2 lines
3. Sysctl is a more flexible way. It provides us with an option when we want to see if skid is existing, we can use sysctl to turn on that.
If you want a sysctl for your own reasons that's fine. But we don't want a sysctl. We want to work without any further configuration.
Quoted 3 lines
Thanks
Jin Yao
- Kyle
Quoted 3 lines · Kyle Huey
If you want a sysctl for your own reasons that's fine. But we don't want a sysctl. We want to work without any further configuration.
Also toggling a sysctl would require root privileges, but rr does not currently need to run as root. Thus rr users would have to either permanently change their system configuration (and every extra configuration step is a pain), or run rr as root so rr can toggle the sysctl itself. Running rr as root is highly undesirable.
Rob
Signature · 5 lines
Su ot deraeppa sah dna Rehtaf eht htiw saw hcihw, efil lanrete eht uoy ot mialcorp ew dna, ti ot yfitset dna ti nees evah ew; deraeppa efil eht. Efil fo Drow eht gninrecnoc mialcorp ew siht - dehcuot evah sdnah ruo dna ta dekool evah ew hcihw, seye ruo htiw nees evah ew hcihw, draeh evah ew hcihw, gninnigeb eht morf saw hcihw taht.
Hi All,
This patch raised many questions, I was prepared. :)
I'd like to try another proposal that it adds a special flag in the returned perf_sample_data to indicate the perf binary that this sample is a leaked sample.
For example, create a new PERF_SAMPLE_RETURN_LEAKAGE in perf_event_sample_format.
In perf_prepare_sample(),
if (event->attr.exclude_kernel && !user_mode(regs))
data->type |= PERF_SAMPLE_RETURN_LEAKAGE;
Now all the samples are kept and the leaked kernel samples are tagged with PERF_SAMPLE_RETURN_LEAKAGE.
In perf binary, it filters out the samples with PERF_SAMPLE_RETURN_LEAKAGE. It needs perf binary modification but rr doesn't need to be changed.
I don't 0-stuffing some fields because:
1. Keeping the skid info should allow us to look at that if we have interesting later.
2. Not sure if 0-stuffing some fields has potential conflicts with some applications.
Is this proposal reasonable?
Thanks
Jin Yao
Quoted 13 lines · Robert O'Callahan
Quoted 3 lines · Kyle Huey
If you want a sysctl for your own reasons that's fine. But we don't want a sysctl. We want to work without any further configuration.
Also toggling a sysctl would require root privileges, but rr does not currently need to run as root. Thus rr users would have to either permanently change their system configuration (and every extra configuration step is a pain), or run rr as root so rr can toggle the sysctl itself. Running rr as root is highly undesirable.
Rob
Quoted 30 lines · Yao
Hi All,
This patch raised many questions, I was prepared. :)
I'd like to try another proposal that it adds a special flag in the returned
perf_sample_datato indicate the perf binary that this sample is a leaked sample.For example, create a new
PERF_SAMPLE_RETURN_LEAKAGEinperf_event_sample_format.In
perf_prepare_sample(),if (
event->attr.exclude_kernel&& !user_mode(regs))data->type |= PERF_SAMPLE_RETURN_LEAKAGE;Now all the samples are kept and the leaked kernel samples are tagged with
PERF_SAMPLE_RETURN_LEAKAGE.In perf binary, it filters out the samples with
PERF_SAMPLE_RETURN_LEAKAGE. It needs perf binary modification but rr doesn't need to be changed.I don't 0-stuffing some fields because:
1. Keeping the skid info should allow us to look at that if we have interesting later.
2. Not sure if 0-stuffing some fields has potential conflicts with some applications.
Quoted 24 lines
Is this proposal reasonable?
Thanks
Jin YaoQuoted 16 lines · Robert O'Callahan
Quoted 4 lines · Kyle Huey
If you want a sysctl for your own reasons that's fine. But we don't want a sysctl. We want to work without any further configuration.
Also toggling a sysctl would require root privileges, but rr does not currently need to run as root. Thus rr users would have to either permanently change their system configuration (and every extra configuration step is a pain), or run rr as root so rr can toggle the sysctl itself. Running rr as root is highly undesirable.
Rob
If the problem you're trying to fix is an inappropriate disclosure of kernel-space information to user-space, how does filtering the samples in user space solve anything?
- Kyle
Quoted 56 lines · Kyle Huey
Quoted 30 lines · Yao
Hi All,
This patch raised many questions, I was prepared. :)
I'd like to try another proposal that it adds a special flag in the returned
perf_sample_datato indicate the perf binary that this sample is a leaked sample.For example, create a new
PERF_SAMPLE_RETURN_LEAKAGEinperf_event_sample_format.In
perf_prepare_sample(),if (
event->attr.exclude_kernel&& !user_mode(regs))data->type |= PERF_SAMPLE_RETURN_LEAKAGE;Now all the samples are kept and the leaked kernel samples are tagged with
PERF_SAMPLE_RETURN_LEAKAGE.In perf binary, it filters out the samples with
PERF_SAMPLE_RETURN_LEAKAGE. It needs perf binary modification but rr doesn't need to be changed.I don't 0-stuffing some fields because:
1. Keeping the skid info should allow us to look at that if we have interesting later.
2. Not sure if 0-stuffing some fields has potential conflicts with some applications.
Quoted 24 lines
Is this proposal reasonable?
Thanks
Jin YaoQuoted 16 lines · Robert O'Callahan
Quoted 4 lines · Kyle Huey
If you want a sysctl for your own reasons that's fine. But we don't want a sysctl. We want to work without any further configuration.
Also toggling a sysctl would require root privileges, but rr does not currently need to run as root. Thus rr users would have to either permanently change their system configuration (and every extra configuration step is a pain), or run rr as root so rr can toggle the sysctl itself. Running rr as root is highly undesirable.
Rob
Quoted 6 lines
If the problem you're trying to fix is an inappropriate disclosure of kernel-space information to user-space, how does filtering the samples in user space solve anything?
- Kyle
In theory it is. But actually we just use perf tool to look at the sampling result.
And suppose a case, if we want to estimate the skid window, we still need the skid info.
Thanks
Jin Yao
Peter Zijlstra
maintainerQuoted 6 lines · Jin Yao
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
Quoted 6 lines
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project.
Now this patch set uses sysctl to control the dropping of leaked kernel samples.
So what happened to the suggestion of keeping the samples but 0-stuffing all the tricky bits?
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Jin Yao
On workloads that do a lot of kernel entry/exits we see kernel samples, even though :u is specified. This is due to skid existing.
This might be a security issue because it can leak kernel addresses even though kernel sampling support is disabled.
One patch "perf/core: Drop kernel samples even though :u is specified" was posted in last year but it was reverted because it introduced a regression issue that broke the rr-project.
Quoted 6 lines
Quoted 2 lines
Now this patch set uses sysctl to control the dropping of leaked kernel samples.
So what happened to the suggestion of keeping the samples but 0-stuffing all the tricky bits?
Bring more overhead to kernel if we zero the bits considering the number of leaked samples may be not too small?
And the skid information may be interesting (see example of hitting on page_fault in previous mail). If we zero it, we will not know.
Thanks
Jin Yao
Peter Zijlstra
maintainerQuoted 2 lines · Yao
Bring more overhead to kernel if we zero the bits considering the number of leaked samples may be not too small?
Keeping the samples at least allows you to know how many samples happened and such things.
Quoted 2 lines
And the skid information may be interesting (see example of hitting on
page_faultin previous mail). If we zero it, we will not know.
If you throw them out the window you also don't know, do you?
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Yao
Bring more overhead to kernel if we zero the bits considering the number of leaked samples may be not too small?
Keeping the samples at least allows you to know how many samples happened and such things.
Yeah, agree, but a little bit overhead ...
Quoted 5 lines
Quoted 2 lines
And the skid information may be interesting (see example of hitting on
page_faultin previous mail). If we zero it, we will not know.If you throw them out the window you also don't know, do you?
Yes, default I can't know that window. I have to use sysctl to enable.
Thanks
Jin Yao
Hi,
Quoted 9 lines · Peter Zijlstra
Quoted 2 lines · Yao
Bring more overhead to kernel if we zero the bits considering the number of leaked samples may be not too small?
Keeping the samples at least allows you to know how many samples happened and such things.
Yes, important point. You cannot just drop a sample. It may break some calculations
done like num_samples x fixed_periods = total number of events observed. Some tools do this.
Quoted 4 lines
Quoted 2 lines
And the skid information may be interesting (see example of hitting on
page_faultin previous mail). If we zero it, we will not know.If you throw them out the window you also don't know, do you?