audit: accelerate audit rule filter
Quoted 96 lines · zhaozixuan (C)
Quoted 36 lines
Quoted 36 lines · Zixuan Zhao
Quoted 34 lines
We used
lat_syscallof lmbench3 to test the performance impact of this patch. We changed the number of rules and runlat_syscallwith 1000 repetitions at each test. Syscalls measured bylat_syscallare not monitored by rules.Before this optimization:
null read write stat fstat open
0 rules 1.87ms 2.74ms 2.56ms 26.31ms 4.13ms 69.66ms 10 rules 2.15ms 3.13ms 3.32ms 26.99ms 4.16ms 74.70ms 20 rules 2.45ms 3.97ms 3.82ms 27.05ms 4.60ms 76.35ms 30 rules 2.64ms 4.52ms 3.95ms 30.30ms 4.94ms 78.94ms 40 rules 2.83ms 4.97ms 4.23ms 32.16ms 5.40ms 81.88ms 50 rules 3.00ms 5.30ms 4.84ms 33.49ms 5.79ms 83.20ms 100 rules 4.24ms 9.75ms 7.42ms 37.68ms 6.55ms 93.70ms 160 rules 5.50ms 16.89ms 12.18ms 51.53ms 17.45ms 155.40msAfter this optimization:
null read write stat fstat open
0 rules 1.81ms 2.84ms 2.42ms 27.70ms 4.15ms 69.10ms 10 rules 1.97ms 2.83ms 2.69ms 27.70ms 4.15ms 69.30ms 20 rules 1.72ms 2.91ms 2.41ms 26.49ms 3.91ms 71.19ms 30 rules 1.85ms 2.94ms 2.48ms 26.27ms 3.97ms 71.43ms 40 rules 1.88ms 2.94ms 2.78ms 26.85ms 4.08ms 69.79ms 50 rules 1.86ms 3.17ms 3.08ms 26.25ms 4.03ms 72.32ms 100 rules 1.84ms 3.00ms 2.81ms 26.25ms 3.98ms 70.25ms 160 rules 1.92ms 3.32ms 3.06ms 26.81ms 4.57ms 71.41msAs the result shown above, the syscall latencies increase as the number of rules increases, while with the patch the latencies remain stable.
This could help when a user adds many audit rules for purposes
(such as attack tracing or process behavior recording) but suffers from low performance.Quoted 8 lines
Quoted 1 line
I have general concerns about trading memory and complexity for performance gains, but beyond that the numbers you posted above don't yet make sense to me.
Thanks for your reply.
The memory cost of this patch is less than 4KB (1820 bytes on x64 and
3640 bytes on compatiblex86_64) which is trivial in many cases. Besides, syscalls are called frequently on a system so a small
optimization could bring a good income.The tradeoff still exists, even though you feel it is worthwhile.
Quoted 43 lines
Quoted 1 line
Why are the latency increases due to rule count not similar across the different syscalls? For example, I would think that if the increase in syscall latency was > >directly attributed to the audit rule processing then the increase on the "open" syscall should be similar to that of the "null" syscall. In other phrasing, if we > >can process 160 rules in ~4ms in the "null" case, why does it take us ~86ms in the "open" case?
As to the test result, we did some investigations and concluded two
reasons:
1. The chosen rule sets were not very suitable. Though they were not hit by syscalls being measured, some of them were hit by other processes, which reduced the system performance and affected the test result; 2. The routine oflat_syscallis much more complicated than we thought. It called many other syscalls during the test, which may cause the result not to be linear.Due to the reasons above, we did another test. We modified audit rule sets and made sure they wouldn't be hit at runtime. Then, we added
ktime_get_real_ts64toauditsc.cto record the time of executing__audit_syscall_exit. We ran "stat" syscall 10000 times for each rule set and recorded the time interval. The result is shown below:Before this optimization:
rule set time
0 rules 3843.96ns1 rules 13119.08ns 10 rules 14003.13ns 20 rules 15420.18ns 30 rules 17284.84ns 40 rules 19010.67ns 50 rules 21112.63ns 100 rules 25815.02ns 130 rules 29447.09nsAfter this optimization:
rule set time
0 rules 3597.78ns1 rules 13498.73ns 10 rules 13122.57ns 20 rules 12874.88ns 30 rules 14351.99ns 40 rules 14181.07ns 50 rules 13806.45ns 100 rules 13890.85ns 130 rules 14441.45nsQuoted 4 lines
As the result showed, the interval is linearly increased before optimization while the interval remains stable after optimization. Note that audit skips some operations if there are no rules, so there is a gap between 0 rule and 1 rule set.
Quoted 5 lines
It looks like a single rule like the one below could effectively disable this optimization, is that correct?
% auditctl -a exit,always -F uid=1001
% auditctl -l
-a always,exit -S all -F uid=1001
Yes, rules like this one which monitors all syscalls could disable the
optimization. The number of the global array could exponentially increase if we want to handle more audit fields. However, we don't that kind of rule is practical because they might generate a great number of logs and even lead to log loss.
Quoted 102 lines · zhaozixuan (C)
Quoted 96 lines · zhaozixuan (C)
Quoted 36 lines
Quoted 36 lines · Zixuan Zhao
Quoted 34 lines
We used
lat_syscallof lmbench3 to test the performance impact of this patch. We changed the number of rules and runlat_syscallwith 1000 repetitions at each test. Syscalls measured bylat_syscallare not monitored by rules.Before this optimization:
null read write stat fstat open
0 rules 1.87ms 2.74ms 2.56ms 26.31ms 4.13ms 69.66ms 10 rules 2.15ms 3.13ms 3.32ms 26.99ms 4.16ms 74.70ms 20 rules 2.45ms 3.97ms 3.82ms 27.05ms 4.60ms 76.35ms 30 rules 2.64ms 4.52ms 3.95ms 30.30ms 4.94ms 78.94ms 40 rules 2.83ms 4.97ms 4.23ms 32.16ms 5.40ms 81.88ms 50 rules 3.00ms 5.30ms 4.84ms 33.49ms 5.79ms 83.20ms 100 rules 4.24ms 9.75ms 7.42ms 37.68ms 6.55ms 93.70ms 160 rules 5.50ms 16.89ms 12.18ms 51.53ms 17.45ms 155.40msAfter this optimization:
null read write stat fstat open
0 rules 1.81ms 2.84ms 2.42ms 27.70ms 4.15ms 69.10ms 10 rules 1.97ms 2.83ms 2.69ms 27.70ms 4.15ms 69.30ms 20 rules 1.72ms 2.91ms 2.41ms 26.49ms 3.91ms 71.19ms 30 rules 1.85ms 2.94ms 2.48ms 26.27ms 3.97ms 71.43ms 40 rules 1.88ms 2.94ms 2.78ms 26.85ms 4.08ms 69.79ms 50 rules 1.86ms 3.17ms 3.08ms 26.25ms 4.03ms 72.32ms 100 rules 1.84ms 3.00ms 2.81ms 26.25ms 3.98ms 70.25ms 160 rules 1.92ms 3.32ms 3.06ms 26.81ms 4.57ms 71.41msAs the result shown above, the syscall latencies increase as the number of rules increases, while with the patch the latencies remain stable.
This could help when a user adds many audit rules for purposes
(such as attack tracing or process behavior recording) but suffers from low performance.Quoted 8 lines
Quoted 1 line
I have general concerns about trading memory and complexity for performance gains, but beyond that the numbers you posted above don't yet make sense to me.
Thanks for your reply.
The memory cost of this patch is less than 4KB (1820 bytes on x64 and
3640 bytes on compatiblex86_64) which is trivial in many cases. Besides, syscalls are called frequently on a system so a small
optimization could bring a good income.The tradeoff still exists, even though you feel it is worthwhile.
Quoted 43 lines
Quoted 1 line
Why are the latency increases due to rule count not similar across the different syscalls? For example, I would think that if the increase in syscall latency was > >directly attributed to the audit rule processing then the increase on the "open" syscall should be similar to that of the "null" syscall. In other phrasing, if we > >can process 160 rules in ~4ms in the "null" case, why does it take us ~86ms in the "open" case?
As to the test result, we did some investigations and concluded two
reasons:
1. The chosen rule sets were not very suitable. Though they were not hit by syscalls being measured, some of them were hit by other processes, which reduced the system performance and affected the test result; 2. The routine oflat_syscallis much more complicated than we thought. It called many other syscalls during the test, which may cause the result not to be linear.Due to the reasons above, we did another test. We modified audit rule sets and made sure they wouldn't be hit at runtime. Then, we added
ktime_get_real_ts64toauditsc.cto record the time of executing__audit_syscall_exit. We ran "stat" syscall 10000 times for each rule set and recorded the time interval. The result is shown below:Before this optimization:
rule set time
0 rules 3843.96ns1 rules 13119.08ns 10 rules 14003.13ns 20 rules 15420.18ns 30 rules 17284.84ns 40 rules 19010.67ns 50 rules 21112.63ns 100 rules 25815.02ns 130 rules 29447.09nsAfter this optimization:
rule set time
0 rules 3597.78ns1 rules 13498.73ns 10 rules 13122.57ns 20 rules 12874.88ns 30 rules 14351.99ns 40 rules 14181.07ns 50 rules 13806.45ns 100 rules 13890.85ns 130 rules 14441.45nsQuoted 4 lines
As the result showed, the interval is linearly increased before optimization while the interval remains stable after optimization. Note that audit skips some operations if there are no rules, so there is a gap between 0 rule and 1 rule set.
Quoted 5 lines
It looks like a single rule like the one below could effectively disable this optimization, is that correct?
% auditctl -a exit,always -F uid=1001
% auditctl -l
-a always,exit -S all -F uid=1001
Quoted 5 lines
Yes, rules like this one which monitors all syscalls could disable the
optimization. The number of the global array could exponentially increase if we want to handle more audit fields. However, we don't that kind of rule is practical because they might generate a great number of logs and even lead to log loss.
Before we merge something like this I think we need a better understand of typical audit filter rules used across the different audit use cases. This patch is too much of a band-aid to merge without a really good promise that it will help most of the real world audit deployments.
Signature · 2 lines
paul moore www.paul-moore.com
Quoted 91 lines · Paul Moore
Quoted 84 lines · zhaozixuan (C)
Quoted 79 lines
Quoted 36 lines · zhaozixuan (C)
Quoted 32 lines
Quoted 24 lines · Zixuan Zhao
We used
lat_syscallof lmbench3 to test the performance impact ofthis patch. We changed the number of rules and runlat_syscallwith1000 repetitions at each test. Syscalls measured bylat_syscallarenot monitored by rules.
Before this optimization:
null read write stat fstat open 0rules 1.87ms 2.74ms 2.56ms 26.31ms 4.13ms 69.66ms 10 rules 2.15ms 3.13ms 3.32ms 26.99ms 4.16ms 74.70ms 20 rules 2.45ms 3.97ms 3.82ms 27.05ms 4.60ms 76.35ms 30 rules 2.64ms 4.52ms 3.95ms 30.30ms 4.94ms 78.94ms 40 rules 2.83ms 4.97ms 4.23ms 32.16ms 5.40ms 81.88ms 50 rules 3.00ms 5.30ms 4.84ms 33.49ms 5.79ms 83.20ms100 rules 4.24ms 9.75ms 7.42ms 37.68ms 6.55ms 93.70ms160 rules 5.50ms 16.89ms 12.18ms 51.53ms 17.45ms 155.40ms After this optimization: null read write stat fstat open 0 rules 1.81ms 2.84ms 2.42ms 27.70ms 4.15ms 69.10ms 10 rules 1.97ms 2.83ms 2.69ms 27.70ms 4.15ms 69.30ms 20 rules 1.72ms 2.91ms 2.41ms 26.49ms 3.91ms 71.19ms 30 rules 1.85ms 2.94ms 2.48ms 26.27ms 3.97ms 71.43ms 40 rules 1.88ms 2.94ms 2.78ms 26.85ms 4.08ms 69.79ms 50 rules 1.86ms 3.17ms 3.08ms 26.25ms 4.03ms 72.32ms100 rules 1.84ms 3.00ms 2.81ms 26.25ms 3.98ms 70.25ms160 rules 1.92ms 3.32ms 3.06ms 26.81ms 4.57ms 71.41msQuoted 5 lines
As the result shown above, the syscall latencies increase as thenumber of rules increases, while with the patch the latencies remain stable. This could help when a user adds many audit rules for purposes(such as attack tracing or process behavior recording) but suffersfrom low performance.
Quoted 3 lines
I have general concerns about trading memory and complexity for performance gains, but beyond that the numbers you posted above don't yet make sense to me.
Quoted 5 lines
Thanks for your reply.
The memory cost of this patch is less than 4KB (1820 bytes on x64 and 3640 bytes on compatiblex86_64) which is trivial in many cases. Besides, syscalls are called frequently on a system so a smalloptimization could bring a good income.The tradeoff still exists, even though you feel it is worthwhile.
Quoted 29 lines
Quoted 6 lines
Why are the latency increases due to rule count not similar across the different syscalls? For example, I would think that if the increase in syscall latency was > >directly attributed to the audit rule processing then the increase on the "open" syscall should be similar to that of the "null" syscall. In other phrasing, if we > >can process 160 rules in ~4ms in the "null" case, why does it take us ~86ms in the "open" case?
As to the test result, we did some investigations and concluded two reasons:1. The chosen rule sets were not very suitable. Though they were nothit by syscalls being measured, some of them were hit by otherprocesses, which reduced the system performance and affected the testresult; 2. The routine of
lat_syscallis much more complicated than wethought. It called many other syscalls during the test, which maycause the result not to be linear.
Due to the reasons above, we did another test. We modified audit rulesets and made sure they wouldn't be hit at runtime. Then, we addedktime_get_real_ts64toauditsc.cto record the time ofexecuting__audit_syscall_exit. We ran "stat" syscall 10000 times for each ruleset and recorded the time interval. The result is shown below:
Before this optimization:
rule set time 0 rules 3843.96ns 1 rules 13119.08ns 10rules 14003.13ns 20 rules 15420.18ns 30 rules 17284.84ns 40 rules 19010.67ns 50 rules 21112.63ns100 rules 25815.02ns130 rules 29447.09ns After this optimization: rule set time 0 rules 3597.78ns 1 rules 13498.73ns 10 rules 13122.57ns 20 rules 12874.88ns 30 rules 14351.99ns 40 rules 14181.07ns 50 rules 13806.45ns100 rules 13890.85ns130 rules 14441.45nsQuoted 4 lines
As the result showed, the interval is linearly increased beforeoptimization while the interval remains stable after optimization.Note that audit skips some operations if there are no rules, so thereis a gap between 0 rule and 1 rule set.
Quoted 4 lines
It looks like a single rule like the one below could effectively disable this optimization, is that correct?
% auditctl -a exit,always -F uid=1001 % auditctl -l -a always,exit -S all
-F uid=1001Quoted 5 lines
Yes, rules like this one which monitors all syscalls could disable the optimization. The number of the global array could exponentially increase if we want to handle more audit fields. However, we don't that kind of rule is practical because they might generate a great number of logs and even lead to log loss.
Quoted 4 lines
Before we merge something like this I think we need a betterunderstand of typical audit filter rules used across the differentaudit use cases. This patch is too much of a band-aid to mergewithout a really good promise that it will help most of the real worldaudit deployments.
For a 'real world deployment, I suggestcd /usr/share/audit/sample-rules cp 10-base-config.rules 11-loginuid.rules 12-ignore-error.rules 30-stig.rules 41- containers.rules 43-module-load.rules 71-networking.rules /etc/audit/rules.d/ rm -f /etc/audit/rules.d/audit.rules # Remove default ruleset if not applicable
echo '-b 32768' > /etc/audit/rules.d/zzexecve.rules
echo '-a exit,always -F arch=b32 -F auid!=2147483647 -S execve -k cmds' >> /etc/audit/rules.d/zzexecve.rules
echo '-a exit,always -F arch=b64 -F auid!=4294967295 -S execve -k cmds' >> /etc/audit/rules.d/zzexecve.rules
--
Linux-audit mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-audit