audit: speed up syscall rule match while exiting syscall
RFC 1/1 audit: speed up syscall rule match while exiting syscall
From 82ebcf43481be21ee3e32ec1749b42f651737880 Mon Sep 17 00:00:00 2001
From: Yang Yang <[email protected]>
Date: Wed, 13 Jan 2021 20:18:04 +0800
Subject: [PATCH] [RFC,v1,1/1] speed up syscall rule match while exiting syscall
If user add any syscall rule, in all syscalls, audit_filter_syscall() traverses struct list_head audit_filter_list to find out whether current syscall match one rule. This takes o(n), which is not necessary, specially for user who add a very few syscall rules. On the other hand, user may not much care about rule add/delete speed. So do o(n) calculate at rule changing, and ease the burden of audit_filter_syscall().
Define audit_syscall[NR_syscalls], every element stands for one syscall. audit_filter_syscall() checks audit_syscall[NR_syscalls]. audit_syscall[n] == 0 indicates no rule audit syscall n, do a quick exit. audit_syscall[n] > 0 indicates at least one rule audit syscall n. audit_syscall[n] update when syscall rule changes.
--- include/linux/audit.h | 2 ++ kernel/audit.c | 2 ++ kernel/auditfilter.c | 16 ++++++++++++++++ kernel/auditsc.c | 9 ++++++++- 4 files changed, 28 insertions(+), 1 deletion(-)
include/linux/audit.h +2 -0
@@ -549,6 +549,8 @@ static inline void audit_log_nfcfg(const char *name, u8 af,extern int audit_n_rules;extern int audit_signals; +extern u32 audit_syscall[NR_syscalls]; +extern int audit_in_mask (const struct audit_krule *rule,unsigned long val);#else /* CONFIG_AUDITSYSCALL */ static inline int audit_alloc (struct task_struct *task) {
kernel/audit.c +2 -0
@@ -208,6 +208,8 @@ struct audit_reply {struct sk_buff *skb; }; +u32 audit_syscall[NR_syscalls] = {0 }; +/** * auditd_test_task - Check to see if a given task is an audit daemon * @task: the task to check
kernel/auditfilter.c +16 -0
@@ -936,6 +936,7 @@ static inline int audit_add_rule(struct audit_entry *entry)int err =0 ;#ifdef CONFIG_AUDITSYSCALL int dont_count =0 ; +int i =0 ;/* If any of these, don't count towards total */ switch (entry->rule.listnr) {
@@ -957,6 +958,13 @@ static inline int audit_add_rule(struct audit_entry *entry)return err; } +#ifdef CONFIG_AUDITSYSCALL +if (entry->rule.listnr ==AUDIT_FILTER_EXIT && !watch && !tree) { +for (i =0 ; i < NR_syscalls; i++) + audit_syscall[i] += !(!audit_in_mask (&entry->rule, i)); + } +#endif +if (watch) {/* audit_filter_mutex is dropped and re-taken during this call */ err =audit_add_watch (&entry->rule, &list);
@@ -1018,6 +1026,7 @@ int audit_del_rule(struct audit_entry *entry)int ret =0 ;#ifdef CONFIG_AUDITSYSCALL int dont_count =0 ; +int i =0 ;/* If any of these, don't count towards total */ switch (entry->rule.listnr) {
@@ -1035,6 +1044,13 @@ int audit_del_rule(struct audit_entry *entry) goto out; } +#ifdef CONFIG_AUDITSYSCALL +if (entry->rule.listnr ==AUDIT_FILTER_EXIT && !e->rule.watch && !e->rule.tree) { +for (i =0 ; i < NR_syscalls; i++) + audit_syscall[i] -= !(!audit_in_mask (&entry->rule, i)); + } +#endif +if (e->rule.watch)audit_remove_watch_rule (&e->rule);
kernel/auditsc.c +8 -1
@@ -783,7 +783,7 @@ static enum audit_state audit_filter_task(struct task_struct *tsk, char **key)return AUDIT_BUILD_CONTEXT ; } -static int audit_in_mask (const struct audit_krule *rule,unsigned long val) +int audit_in_mask (const struct audit_krule *rule,unsigned long val) {int word, bit;
@@ -814,6 +814,13 @@ static enum audit_state audit_filter_syscall(struct task_struct *tsk,if (auditd_test_task (tsk))return AUDIT_DISABLED ; +mutex_lock (&audit_filter_mutex); +if (!audit_syscall[ctx->major]) { +mutex_unlock (&audit_filter_mutex); +return AUDIT_BUILD_CONTEXT ; + } +mutex_unlock (&audit_filter_mutex); +rcu_read_lock ();list_for_each_entry_rcu (e, list, list) { if (audit_in_mask(&e->rule, ctx->major) &&
--
2.15.2
--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit
Paul Moore
maintainerQuoted 17 lines · On Wed, Jan 13, 2021 at 7:39 AM <[email protected]> wrote:
From
82ebcf43481be21ee3e32ec1749b42f651737880Mon Sep 17 00:00:00 2001
From: Yang Yang <[email protected]>
Date: Wed, 13 Jan 2021 20:18:04 +0800
Subject: [PATCH] [RFC,v1,1/1] speed up syscall rule match while exiting syscall
If user add any syscall rule, in all syscalls,audit_filter_syscall()traverses structlist_headaudit_filter_listto find out whether current syscall match one rule. This takes o(n), which is not necessary, specially for user who add a very few syscall rules. On the other hand, user may not much care about rule add/delete speed. So do o(n) calculate at rule changing, and ease the burden ofaudit_filter_syscall().Define audit_syscall[NR_syscalls], every element stands for one syscall.
audit_filter_syscall()checks audit_syscall[NR_syscalls]. audit_syscall[n] == 0 indicates no rule audit syscall n, do a quick exit. audit_syscall[n] > 0 indicates at least one rule audit syscall n. audit_syscall[n] update when syscall rule changes.
Quoted 7 lines
Signed-off-by Yang Yang <[email protected]>--- include/linux/audit.h | 2 ++ kernel/audit.c | 2 ++ kernel/auditfilter.c | 16 ++++++++++++++++ kernel/auditsc.c | 9 ++++++++- 4 files changed, 28 insertions(+), 1 deletion(-)
Before we go too far into a review of this patch, please provide some performance measurements using a variety of rule counts, both common and extreme, so that we can better judge the benefits of this patch. The measurements should include both the rule add/delete time deltas as well as the impact on the syscall invocations. If non-obvious, please also include how you performed the measurements and captured the data.
These are good things to include in the commit description when submitting patches focused on improving performance.
Signature · 7 lines
paul moore www.paul-moore.com -- Linux-audit mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-audit
Performance measurements:
1.Environment
CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Linux kernel version: 5.11-rc3
Audit version: 2.8.4
2.Result
2.1 Syscall invocations
Test method:
Run command "top" with no-load.
Add rule likes "auditctl -a always,exit -F arch=b64 -S chmod -F auid=[number]" which doesn't hit audit. User command "perf record -Rg -t [top's pid] sleep 900" to get audit_filter_syscall()'s execute time ratio.
audit_filter_syscall() ratio with 100 rules:
before this patch: 15.29%.
after this patch: 0.88%, reduce 14.41%.
audit_filter_syscall() ratio with CIS[1] rules:
before this patch: 2.25%.
after this patch: 1.93%%, reduce 0.32%.
audit_filter_syscall() ratio with 10 rules:
before this patch: 0.94%.
after this patch: 1.02%, increase 0.08%.
audit_filter_syscall() ratio with 1 rule:
before this patch: 0.20%.
after this patch: 0.88%, increase 0.68%.Analyse:
With 1 rule, after this patch performance is worse, because mutex_lock()/mutex_unlock(). But user just add one rule seems unusual. With more rule, after this patch performance is improved.Typical likes CIS benchmark.
2.2 Rule change
Test method:
Use ktime_get_real_ts64() before and after audit_add_rule()/audit_del_rule() to calculate time.
Add/delete rule by command "auditctl". Each test 10times and get average.
audit_add_rule() time:
before this patch: 3120ns.
after this patch: 7783ns, increase 149%.audit_del_rule() time:
before this patch: 3510ns.
after this patch: 8519ns, increase 143%.
Analyse:
After this patch, rule change time obviously increase. But rule change may not happen very often.
[1] CIS is a Linux Benchmarks for security purpose.
https://www.cisecurity.org/benchmark/distribution_independent_linux/
------------------Original Mail------------------
Sender: PaulMoore
To: yang yang10192021;
CC: Eric Paris;[email protected];[email protected];
Date: 2021/01/13 21:48
Subject: Re:[RFC,v1,1/1] audit: speed up syscall rule match while exiting syscall
Quoted 17 lines · On Wed, Jan 13, 2021 at 7:39 AM <[email protected]> wrote:
From
82ebcf43481be21ee3e32ec1749b42f651737880Mon Sep 17 00:00:00 2001
From: Yang Yang <[email protected]>
Date: Wed, 13 Jan 2021 20:18:04 +0800
Subject: [PATCH] [RFC,v1,1/1] speed up syscall rule match while exiting syscall
If user add any syscall rule, in all syscalls,audit_filter_syscall()traverses structlist_headaudit_filter_listto find out whether current syscall match one rule. This takes o(n), which is not necessary, specially for user who add a very few syscall rules. On the other hand, user may not much care about rule add/delete speed. So do o(n) calculate at rule changing, and ease the burden ofaudit_filter_syscall().Define audit_syscall[NR_syscalls], every element stands for one syscall.
audit_filter_syscall()checks audit_syscall[NR_syscalls]. audit_syscall[n] == 0 indicates no rule audit syscall n, do a quick exit. audit_syscall[n] > 0 indicates at least one rule audit syscall n. audit_syscall[n] update when syscall rule changes.
Quoted 7 lines
Signed-off-by Yang Yang <[email protected]>--- include/linux/audit.h | 2 ++ kernel/audit.c | 2 ++ kernel/auditfilter.c | 16 ++++++++++++++++ kernel/auditsc.c | 9 ++++++++- 4 files changed, 28 insertions(+), 1 deletion(-)
Before we go too far into a review of this patch, please provide some performance measurements using a variety of rule counts, both common and extreme, so that we can better judge the benefits of this patch. The measurements should include both the rule add/delete time deltas as well as the impact on the syscall invocations. If non-obvious, please also include how you performed the measurements and captured the data.
These are good things to include in the commit description when submitting patches focused on improving performance.
--
paul moore
www.paul-moore.com
--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit
Paul Moore
maintainerQuoted 7 lines · On Thu, Jan 14, 2021 at 8:25 AM <[email protected]> wrote:
Performance measurements:
1.Environment
CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Linux kernel version: 5.11-rc3
Audit version: 2.8.4
Quoted 6 lines
2.Result
2.1 Syscall invocations
Test method:
Run command "top" with no-load.
Add rule likes "auditctl -a always,exit -F arch=b64 -S chmod -F auid=[number]" which doesn't hit audit. User command "perf record -Rg -t [top's pid] sleep 900" to get audit_filter_syscall()'s execute time ratio.
Thanks for providing some performance numbers so quickly, a few comments and thoughts below ...
Quoted 12 lines
audit_filter_syscall()ratio with 100 rules:before this patch: 15.29%. after this patch: 0.88%, reduce 14.41%. audit_filter_syscall() ratio with CIS[1] rules: before this patch: 2.25%. after this patch: 1.93%%, reduce 0.32%. audit_filter_syscall() ratio with 10 rules: before this patch: 0.94%. after this patch: 1.02%, increase 0.08%. audit_filter_syscall() ratio with 1 rule: before this patch: 0.20%. after this patch: 0.88%, increase 0.68%.
If we assume the CIS rules to be a reasonable common case (I'm not sure if that is correct or not, but we'll skip that discussion for now), we see an performance improvement of 0.32% correct, yes? We also see a performance regression with small number of syscall rules that equalizes above ten rules, yes?
On your system can you provide some absolute numbers? For example, what does 0.32% equate to in terms of wall clock time for a given syscall invocation?
Quoted 8 lines
Analyse:
With 1 rule, after this patch performance is worse, because mutex_lock()/mutex_unlock(). But user just add one rule seems unusual. With more rule, after this patch performance is improved.Typical likes CIS benchmark.2.2 Rule change
Test method:
Usektime_get_real_ts64()before and after audit_add_rule()/audit_del_rule() to calculate time.
Add/delete rule by command "auditctl". Each test 10times and get average.
In this case I'm less concerned about micro benchmarks, and more interested in the wall clock time difference when running auditctl to add/remove rules. The difference here in the micro benchmark is not trivial, but with a delta of 4~5us it is possible that it is a small(er) percentage when compared to the total time spent executing auditctl.
Quoted 12 lines
audit_add_rule()time:
before this patch: 3120ns.
after this patch: 7783ns, increase 149%.audit_del_rule()time:
before this patch: 3510ns.
after this patch: 8519ns, increase 143%.Analyse:
After this patch, rule change time obviously increase. But rule change may not happen very often.[1] CIS is a Linux Benchmarks for security purpose.
https://www.cisecurity.org/benchmark/distribution_independent_linux/
Signature · 7 lines
paul moore www.paul-moore.com -- Linux-audit mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-audit
RFC v2 1/1 audit: speed up syscall rule match while exiting syscall
From 72f3ecde58edb03d76cb359607fef98c1663d481 Mon Sep 17 00:00:00 2001
From: Yang Yang <[email protected]>
Date: Thu, 21 Jan 2021 21:05:04 +0800
Subject: [PATCH] [RFC,v2,1/1] speed up syscall rule match while exiting syscall
audit_filter_syscall() traverses struct list_head audit_filter_list to find out whether current syscall match one rule. This takes o(n), which is not necessary, specially for user who add a very few syscall rules. On the other hand, user may not much care about rule add/delete speed. So do o(n) calculate at rule changing, and ease the burden of audit_filter_syscall().
Define audit_syscall[NR_syscalls], every element stands for one syscall. audit_filter_syscall() checks audit_syscall[NR_syscalls]. audit_syscall[n] == 0 indicates no rule audit syscall n, do a quick exit. audit_syscall[n] > 0 indicates at least one rule audit syscall n. audit_syscall[n] update when syscall rule changes.
--- include/linux/audit.h | 2 ++ kernel/audit.c | 4 ++++ kernel/auditfilter.c | 30 ++++++++++++++++++++++++++++++ kernel/auditsc.c | 5 ++++- 4 files changed, 40 insertions(+), 1 deletion(-)
include/linux/audit.h +2 -0
@@ -549,6 +549,8 @@ static inline void audit_log_nfcfg(const char *name, u8 af,extern int audit_n_rules;extern int audit_signals; +extern u32 auditing_syscall[NR_syscalls]; +extern int audit_in_mask (const struct audit_krule *rule,unsigned long val);#else /* CONFIG_AUDITSYSCALL */ static inline int audit_alloc (struct task_struct *task) {
kernel/audit.c +4 -0
@@ -208,6 +208,10 @@ struct audit_reply {struct sk_buff *skb; }; +#ifdef CONFIG_AUDITSYSCALL +u32 auditing_syscall[NR_syscalls] = {0 }; +#endif +/** * auditd_test_task - Check to see if a given task is an audit daemon * @task: the task to check
kernel/auditfilter.c +30 -0
@@ -926,6 +926,28 @@ static struct audit_entry *audit_find_rule(struct audit_entry *entry,static u64 prio_low = ~0ULL /2 ;static u64 prio_high = ~0ULL /2 -1 ; +#ifdef CONFIG_AUDITSYSCALL +static inline void update_auditing_syscall (struct audit_krule rule,bool add) +{ +int i; + +/* syscall rule with type AUDIT_FILTER_EXIT */ +if (rule.listnr ==AUDIT_FILTER_EXIT && !rule.watch && !rule.tree) { +for (i =0 ; i < NR_syscalls; i++) { +/* whether this rule include one syscall */ +if (unlikely (audit_in_mask (&rule, i))) { +if (add == true) + auditing_syscall[i]++; +else + auditing_syscall[i]--; + } + } + } + +return ; +} +#endif +/* Add rule to given filterlist if not a duplicate. */ static inline int audit_add_rule (struct audit_entry *entry) {
@@ -957,6 +979,10 @@ static inline int audit_add_rule(struct audit_entry *entry)return err; } +#ifdef CONFIG_AUDITSYSCALL +update_auditing_syscall (entry->rule, true); +#endif +if (watch) {/* audit_filter_mutex is dropped and re-taken during this call */ err =audit_add_watch (&entry->rule, &list);
@@ -1035,6 +1061,10 @@ int audit_del_rule(struct audit_entry *entry) goto out; } +#ifdef CONFIG_AUDITSYSCALL +update_auditing_syscall (e->rule, false); +#endif +if (e->rule.watch)audit_remove_watch_rule (&e->rule);
kernel/auditsc.c +4 -1
@@ -783,7 +783,7 @@ static enum audit_state audit_filter_task(struct task_struct *tsk, char **key)return AUDIT_BUILD_CONTEXT ; } -static int audit_in_mask (const struct audit_krule *rule,unsigned long val) +int audit_in_mask (const struct audit_krule *rule,unsigned long val) {int word, bit;
@@ -814,6 +814,9 @@ static enum audit_state audit_filter_syscall(struct task_struct *tsk,if (auditd_test_task (tsk))return AUDIT_DISABLED ; +if (!auditing_syscall[ctx->major]) +return AUDIT_BUILD_CONTEXT ; +rcu_read_lock ();list_for_each_entry_rcu (e, list, list) { if (audit_in_mask(&e->rule, ctx->major) &&
Signature · 1 line
2.15.2
--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit
Paul Moore
maintainerQuoted 24 lines · On Thu, Jan 21, 2021 at 8:54 AM <[email protected]> wrote:
From
72f3ecde58edb03d76cb359607fef98c1663d481Mon Sep 17 00:00:00 2001
From: Yang Yang <[email protected]>
Date: Thu, 21 Jan 2021 21:05:04 +0800
Subject: [PATCH] [RFC,v2,1/1] speed up syscall rule match while exiting syscall
audit_filter_syscall()traverses structlist_headaudit_filter_listto find out whether current syscall match one rule. This takes o(n), which is not necessary, specially for user who add a very few syscall rules. On the other hand, user may not much care about rule add/delete speed. So do o(n) calculate at rule changing, and ease the burden ofaudit_filter_syscall().Define audit_syscall[NR_syscalls], every element stands for one syscall.
audit_filter_syscall()checks audit_syscall[NR_syscalls]. audit_syscall[n] == 0 indicates no rule audit syscall n, do a quick exit. audit_syscall[n] > 0 indicates at least one rule audit syscall n. audit_syscall[n] update when syscall rule changes.Signed-off-by Yang Yang <[email protected]>--- include/linux/audit.h | 2 ++ kernel/audit.c | 4 ++++ kernel/auditfilter.c | 30 ++++++++++++++++++++++++++++++ kernel/auditsc.c | 5 ++++- 4 files changed, 40 insertions(+), 1 deletion(-)
...
Quoted 34 lines
kernel/auditfilter.c +22
-0@@ -926,6 +926,28 @@ static struct audit_entry *audit_find_rule(struct audit_entry *entry,static u64 prio_low = ~0ULL /2 ;static u64 prio_high = ~0ULL /2 -1 ; +#ifdef CONFIG_AUDITSYSCALL +static inline void update_auditing_syscall (struct audit_krule rule,bool add) +{ +int i; + +/* syscall rule with type AUDIT_FILTER_EXIT */ +if (rule.listnr ==AUDIT_FILTER_EXIT && !rule.watch && !rule.tree) { +for (i =0 ; i < NR_syscalls; i++) { +/* whether this rule include one syscall */ +if (unlikely (audit_in_mask (&rule, i))) { +if (add == true) + auditing_syscall[i]++; +else + auditing_syscall[i]--; + } + } + } + +return ; +} +#endif +/* Add rule to given filterlist if not a duplicate. */ static inline int audit_add_rule (struct audit_entry *entry) {
kernel/auditfilter.c static inline int audit_add_rule(struct audit_entry *entry) · line 984
Quoted 6 lines
kernel/auditfilter.c +3
-0@@ -957,6 +979,10 @@ static inline int audit_add_rule(struct audit_entry *entry)return err; } +#ifdef CONFIG_AUDITSYSCALL +update_auditing_syscall (entry->rule, true); +#endif
I'm going to reply to your other email where we are discussing the performance of this patch, but I wanted to make one comment about the approach you've taken with the update_auditing_syscall() here.
First, naming things is hard, but the chosen name is not a good one in my opinion. Something like audit_rule_syscall_mask_update() would probably be a better fit.
Second, in order to minimize preprocessor clutter, it is better to use the following pattern:
#ifdef CONFIG_FOO
int func(int arg)
{
/* important stuff */
}
#else
int func(int arg)
{
return 0; /* appropriate return value */
}
#endifThere are probably a few other comments on this patch, but I want us to discuss the performance impacts of this first as I'm not convinced this is a solution we want upstream.
Signature · 7 lines
paul moore www.paul-moore.com -- Linux-audit mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-audit
Thanks for reply, I have sent a new patch with better performance. The v1 patch uses mutex() is not necessary.
Performance measurements:
1.Environment
CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Linux kernel version: 5.11-rc3
Audit version: 2.8.4
2.Result
2.1 Syscall ratio
Run command "top" with no-load.
Add rule likes "auditctl -a always,exit -F arch=b64 -S chmod -F auid=[number]" which doesn't hit audit. User command "perf record -Rg -t [top's pid] sleep 900" to get audit_filter_syscall()'s execute time ratio.
audit_filter_syscall() ratio with 100 rules:
before this patch: 15.29%.
after this patch: 0.13%, reduce 15.16%.
audit_filter_syscall() ratio with CIS rules:
before this patch: 2.25%.
after this patch: 1.21%%, reduce 1.04%.
audit_filter_syscall() ratio with 10 rules:
before this patch: 0.94%.
after this patch: 0.16%, reduce 0.78%.
audit_filter_syscall() ratio with 1 rule:
before this patch: 0.20%.
after this patch: 0.04%, reduce 0.16%.
Analyse:
Performance is better overall.
2.2 Syscall absolute time
Test method:
Use ktime_get_real_ts64() in do_syscall_64() to calculate time.
Run command "chmod 777 /etc/fstab" with chown rules. Each test 10times and get average.
do_syscall_64() time with 100 rules:
before this patch: 7604ns
after this patch: 5244ns, reduce 2360ns.
do_syscall_64() time with CIS rules:
before this patch: 6710ns
after this patch: 7293ns, increase 583ns.
do_syscall_64() time with 10 rules:
before this patch: 5382ns
after this patch: 5171ns, reduce 211ns.
do_syscall_64() time with 1 rule:
before this patch: 5361ns
after this patch: 5375ns, increase 14ns.
do_syscall_64() time with no rules:
before this patch: 4735ns
after this patch: 4804ns, increase 69ns.
Analyse:
With a few rules, performance is close.
With 100 rules, performance is better, but with CIS rules performance regress. Maybe relevant to certain syscall.
2.3 Rule change absolute time
Test method:
Use command "time ..." to get rule change absolute time.
add 1 rule: time auditctl -a always,exit -F arch=b64 -S chmod -F auid=1 before this patch:
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
after this patch:
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00sdelete 1 rule: time auditctl -d always,exit -F arch=b64 -S chmod -F auid=1 before this patch:
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
after this patch:
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00sadd 100 rule: time augenrules --load
before this patch:
real 0m 0.01s
user 0m 0.00s
sys 0m 0.00s
after this patch:
real 0m 0.02s
user 0m 0.01s
sys 0m 0.00sdelete 100 rule:time auditctl -D
before this patch:
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
after this patch:
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00sAnalyse:
Performance regress is really trivial.
------------------Original Mail------------------
Sender: PaulMoore
To: yang yang10192021;
CC: Eric Paris;[email protected];[email protected];
Date: 2021/01/14 23:23
Subject: Re:Fw:Re:[RFC,v1,1/1] audit: speed up syscall rule match while exiting syscall
Quoted 7 lines · On Thu, Jan 14, 2021 at 8:25 AM <[email protected]> wrote:
Performance measurements:
1.Environment
CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Linux kernel version: 5.11-rc3
Audit version: 2.8.4
Quoted 6 lines
2.Result
2.1 Syscall invocations
Test method:
Run command "top" with no-load.
Add rule likes "auditctl -a always,exit -F arch=b64 -S chmod -F auid=[number]" which doesn't hit audit. User command "perf record -Rg -t [top's pid] sleep 900" to get audit_filter_syscall()'s execute time ratio.
Thanks for providing some performance numbers so quickly, a few comments and thoughts below ...
Quoted 12 lines
audit_filter_syscall()ratio with 100 rules:before this patch: 15.29%. after this patch: 0.88%, reduce 14.41%. audit_filter_syscall() ratio with CIS[1] rules: before this patch: 2.25%. after this patch: 1.93%%, reduce 0.32%. audit_filter_syscall() ratio with 10 rules: before this patch: 0.94%. after this patch: 1.02%, increase 0.08%. audit_filter_syscall() ratio with 1 rule: before this patch: 0.20%. after this patch: 0.88%, increase 0.68%.
If we assume the CIS rules to be a reasonable common case (I'm not sure if that is correct or not, but we'll skip that discussion for now), we see an performance improvement of 0.32% correct, yes? We also see a performance regression with small number of syscall rules that equalizes above ten rules, yes?
On your system can you provide some absolute numbers? For example, what does 0.32% equate to in terms of wall clock time for a given syscall invocation?
Quoted 8 lines
Analyse:
With 1 rule, after this patch performance is worse, because mutex_lock()/mutex_unlock(). But user just add one rule seems unusual. With more rule, after this patch performance is improved.Typical likes CIS benchmark.2.2 Rule change
Test method:
Usektime_get_real_ts64()before and after audit_add_rule()/audit_del_rule() to calculate time.
Add/delete rule by command "auditctl". Each test 10times and get average.
In this case I'm less concerned about micro benchmarks, and more interested in the wall clock time difference when running auditctl to add/remove rules. The difference here in the micro benchmark is not trivial, but with a delta of 4~5us it is possible that it is a small(er) percentage when compared to the total time spent executing auditctl.
Quoted 7 lines
audit_add_rule()time:
before this patch: 3120ns.
after this patch: 7783ns, increase 149%.audit_del_rule()time:
before this patch: 3510ns.
after this patch: 8519ns, increase 143%.
Quoted 5 lines
Analyse:
After this patch, rule change time obviously increase. But rule change may not happen very often.[1] CIS is a Linux Benchmarks for security purpose.
https://www.cisecurity.org/benchmark/distribution_independent_linux/
--
paul moore
www.paul-moore.com
--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit
Paul Moore
maintainerQuoted 9 lines · On Thu, Jan 21, 2021 at 9:25 AM <[email protected]> wrote:
Thanks for reply, I have sent a new patch with better performance. The v1 patch uses
mutex()is not necessary.Performance measurements:
1.Environment
CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Linux kernel version: 5.11-rc3
Audit version: 2.8.4
...
Quoted 21 lines
2.2 Syscall absolute time
Test method:
Usektime_get_real_ts64()indo_syscall_64()to calculate time.
Run command "chmod 777/etc/fstab" with chown rules. Each test 10times and get average.
do_syscall_64()time with 100 rules:
before this patch: 7604ns
after this patch: 5244ns, reduce 2360ns.
do_syscall_64()time with CIS rules:
before this patch: 6710ns
after this patch: 7293ns, increase 583ns.
do_syscall_64()time with 10 rules:
before this patch: 5382ns
after this patch: 5171ns, reduce 211ns.
do_syscall_64()time with 1 rule:
before this patch: 5361ns
after this patch: 5375ns, increase 14ns.
Quoted 7 lines
do_syscall_64()time with no rules:
before this patch: 4735ns
after this patch: 4804ns, increase 69ns.Analyse:
With a few rules, performance is close.
With 100 rules, performance is better, but with CIS rules performance regress. Maybe relevant to certain syscall.
These numbers aren't particularly good in my opinion, the negative impact of the change to a small number of rules and to the CIS ruleset is not a good thing. It also should be said that you are increasing the memory footprint, even if it is relatively small.
However, if we take a step back and look at the motivation for this work I wonder if there are some things we can do to improve the per-syscall rule processing performance. On thing that jumped out just now was this code in __audit_syscall_exit():
void __audit_syscall_exit(int success, long return_code) {
/* ... */
/*
* we need to fix up the return code in the audit logs if the
* actual return codes are later going to be fixed up by the
* arch specific signal handlers ... */
if (unlikely(return_code <= -ERESTARTSYS) &&
(return_code >= -ERESTART_RESTARTBLOCK) &&
(return_code != -ENOIOCTLCMD))
context->return_code = -EINTR;
else
context->return_code = return_code; audit_filter_syscall(current, context,
&audit_filter_list[AUDIT_FILTER_EXIT]);
audit_filter_inodes(current, context);
if (context->current_state == AUDIT_RECORD_CONTEXT)
audit_log_exit();
}... in the snippet above the audit_filter_inodes() function/rules are given priority over the syscall rules in audit_filter_syscall(AUDIT_FILTER_EXIT), so why not first execute audit_filter_inodes() and only execute audit_filter_syscall(AUDIT_FILTER_EXIT) if necessary? It may be that I'm missing something on this quick look at the code, but I think it is worth investigating. It's also possible there are other similar improvements to made.
There is similar code in __audit_free() but that should be less performance critical.
Signature · 7 lines
paul moore www.paul-moore.com -- Linux-audit mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-audit
Great thanks for your suggestings, patch v3 has a better performance.
New data:
Syscall absolute time
Test method:
Use ktime_get_real_ts64() in do_syscall_64() to calculate time.
Run command "chmod 777 /etc/fstab" with chown rules. Each test 10times and get average.
do_syscall_64() time with 100 rules:
before this patch: 7604ns
after this patch: 5244ns, reduce 2360ns.
do_syscall_64() time with CIS rules:
before this patch: 6710ns
after this patch: 7190ns, increase 480ns.
do_syscall_64() time with CIS rules without chmod rule:
before this patch: 6710ns
after this patch: 5312ns, reduce 1398ns.
do_syscall_64() time with 10 rules:
before this patch: 5382ns
after this patch: 5217ns, reduce 165ns.
do_syscall_64() time with 1 rule:
before this patch: 5361ns
after this patch: 5242ns, reduce 119ns.
do_syscall_64() time with no rules:
before this patch: 4735ns
after this patch: 4804ns, increase 69ns
Analyse:
With no rule, this patch has no effect. The difference data is because of system jitter. And I finger out why with CIS rules this patch has negative impact. Because CIS rules audit syscall chmod, and I test with "chmod syscall". And after I delete the 2 chmod rules in CIS, performance is much better.
Summarize:
If system spend much more time on syscalls that don't need to be audited than syscalls need to be, this patch has obviously improvement.
------------------Original Mail------------------
Sender: PaulMoore
To: yang yang10192021;
CC: [email protected];[email protected];
Date: 2021/01/23 01:31
Subject: Re:Fw:Re:Fw:Re:[RFC,v1,1/1] audit: speed up syscall rule match while exiting syscall
Quoted 9 lines · On Thu, Jan 21, 2021 at 9:25 AM <[email protected]> wrote:
Thanks for reply, I have sent a new patch with better performance. The v1 patch uses
mutex()is not necessary.Performance measurements:
1.Environment
CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Linux kernel version: 5.11-rc3
Audit version: 2.8.4
....
Quoted 21 lines
2.2 Syscall absolute time
Test method:
Usektime_get_real_ts64()indo_syscall_64()to calculate time.
Run command "chmod 777/etc/fstab" with chown rules. Each test 10times and get average.
do_syscall_64()time with 100 rules:
before this patch: 7604ns
after this patch: 5244ns, reduce 2360ns.
do_syscall_64()time with CIS rules:
before this patch: 6710ns
after this patch: 7293ns, increase 583ns.
do_syscall_64()time with 10 rules:
before this patch: 5382ns
after this patch: 5171ns, reduce 211ns.
do_syscall_64()time with 1 rule:
before this patch: 5361ns
after this patch: 5375ns, increase 14ns.
Quoted 7 lines
do_syscall_64()time with no rules:
before this patch: 4735ns
after this patch: 4804ns, increase 69ns.Analyse:
With a few rules, performance is close.
With 100 rules, performance is better, but with CIS rules performance regress. Maybe relevant to certain syscall.
These numbers aren't particularly good in my opinion, the negative impact of the change to a small number of rules and to the CIS ruleset is not a good thing. It also should be said that you are increasing the memory footprint, even if it is relatively small.
However, if we take a step back and look at the motivation for this work I wonder if there are some things we can do to improve the per-syscall rule processing performance. On thing that jumped out just now was this code in __audit_syscall_exit():
void __audit_syscall_exit(int success, long return_code) {
/* ... */
/*
* we need to fix up the return code in the audit logs if the
* actual return codes are later going to be fixed up by the
* arch specific signal handlers ... */
if (unlikely(return_code <= -ERESTARTSYS) && (return_code >= -ERESTART_RESTARTBLOCK) && (return_code != -ENOIOCTLCMD)) context->return_code = -EINTR;
elsecontext->return_code = return_code;
audit_filter_syscall(current, context, &audit_filter_list[AUDIT_FILTER_EXIT]); audit_filter_inodes(current, context);
if (context->current_state == AUDIT_RECORD_CONTEXT) audit_log_exit();
}
.... in the snippet above the audit_filter_inodes() function/rules are given priority over the syscall rules in audit_filter_syscall(AUDIT_FILTER_EXIT), so why not first execute audit_filter_inodes() and only execute audit_filter_syscall(AUDIT_FILTER_EXIT) if necessary? It may be that I'm missing something on this quick look at the code, but I think it is worth investigating. It's also possible there are other similar improvements to made.
There is similar code in __audit_free() but that should be less performance critical.
--
paul moore
www.paul-moore.com
--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit