ntp audit spew.
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offset old=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476
This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Audit isn't even enabled on these machines.
# auditctl -l No rules
# auditctl -s enabled 0 failure 1 pid 0 rate_limit 0 backlog_limit 64 lost 0 backlog 0 loginuid_immutable 0 unlocked
Asides from the log spew, why is this code doing _anything_ when audit isn't enabled ?
Something like this:
kernel/audit.c +3 -0
@@ -2340,6 +2340,9 @@ void audit_log(struct audit_context *ctx, gfp_t gfp_mask, int type,struct audit_buffer *ab;va_list args; +if (audit_initialized !=AUDIT_INITIALIZED ) +return ; + ab =audit_log_start (ctx, gfp_mask, type);if (ab) {va_start (args, fmt);
Might silence the spew, but I'm concerned that the amount of work that audit is doing on an unconfigured machine might warrant further investigation.
("turn off CONFIG_AUDIT" isn't an option unfortunately, as this is a one-size-fits-all kernel that runs on some other hosts that /do/ have audit configured)
Dave
Paul Moore
maintainerQuoted 10 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offset old=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476
Quoted 7 lines
This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Audit isn't even enabled on these machines.
# auditctl -l No rules
[NOTE: added linux-audit to the CC line]
There is an audit mailing list, please CC it when you have audit concerns/questions/etc.
What happens when you run 'auditctl -a never,task'? That *should* silence those messages as the audit_ntp_log() function has the requisite audit_dummy_context() check. FWIW, this is the distro default for many (most? all?) distros; for example, check /etc/audit/audit.rules on a stock Fedora system. A more selective configuration could simply exclude the TIME_ADJNTPVAL record (type 1333) from the records that the kernel emits.
We could also add a audit_enabled check at the top of audit_ntp_log()/__audit_ntp_log(), but I imagine some of that depends on the various security requirements (they can be bizzare and I can't say I'm up to date on all those - Steve Grubb should be able to comment on that).
Signature · 2 lines
paul moore www.paul-moore.com
Quoted 14 lines · Paul Moore
Quoted 13 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offset old=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Quoted 8 lines
Quoted 4 lines
Audit isn't even enabled on these machines.
# auditctl -l No rulesWhat happens when you run 'auditctl -a never,task'? That *should* silence those messages as the
audit_ntp_log()function has the requisiteaudit_dummy_context()check.
They still get emitted.
Quoted 3 lines
FWIW, this is the distro
default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system.
As these machines aren't using audit, they aren't running auditd either.
Essentially: nothing enables audit, but the kernel side continues to log ntp regardless (no other audit messages seem to do this).
Dave
Paul Moore
maintainerQuoted 26 lines · Dave Jones
Quoted 14 lines · Paul Moore
Quoted 13 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offset old=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Quoted 8 lines
Quoted 4 lines
Audit isn't even enabled on these machines.
# auditctl -l No rulesWhat happens when you run 'auditctl -a never,task'? That *should* silence those messages as the
audit_ntp_log()function has the requisiteaudit_dummy_context()check.They still get emitted.
Quoted 7 lines
Quoted 3 lines
FWIW, this is the distro
default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system.As these machines aren't using audit, they aren't running auditd either.
Essentially: nothing enables audit, but the kernel side continues to log ntp regardless (no other audit messages seem to do this).
What does your kernel command line look like? Do you have "audit=1" somewhere in there?
Signature · 2 lines
paul moore www.paul-moore.com
Quoted 31 lines · Paul Moore
Quoted 30 lines · Dave Jones
Quoted 14 lines · Paul Moore
Quoted 13 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offset old=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Quoted 8 lines
Quoted 4 lines
Audit isn't even enabled on these machines.
# auditctl -l No rulesWhat happens when you run 'auditctl -a never,task'? That *should* silence those messages as the
audit_ntp_log()function has the requisiteaudit_dummy_context()check.They still get emitted.
Quoted 3 lines
FWIW, this is the distro
default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system.
Quoted 6 lines
Quoted 3 lines
As these machines aren't using audit, they aren't running auditd either.
Essentially: nothing enables audit, but the kernel side continues to log ntp regardless (no other audit messages seem to do this).What does your kernel command line look like? Do you have "audit=1" somewhere in there?
nope.
ro root=LABEL=/ biosdevname=0 net.ifnames=0 fsck.repair=yes systemd.gpt_auto=0 pcie_pme=nomsi ipv6.autoconf=0 erst_disable crashkernel=128M console=tty0 console=ttyS1,57600 intel_iommu=tboot_noforce
Dave
Eric Paris
maintainerIs this the thing where systemd is listening on the multicast netlink socket and causes everything to come out kmesg as well?
Quoted 63 lines · Dave Jones
Quoted 1 line · Paul Moore
On Mon, Sep 23, 2019 at 12:58 PM Dave Jones <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines · Paul Moore
Quoted 1 line
On Mon, Sep 23, 2019 at 11:50 AM Dave Jones <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I have some hosts that are constantly spewing audit
messages like so:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[46897.591182] audit: type=1333 audit(1569250288.663:220):op=offset old=2543677901372 new=2980866217213
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[46897.591184] audit: type=1333 audit(1569250288.663:221):op=freq old=-2443166611284 new=-2436281764244
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[48850.604005] audit: type=1333 audit(1569252241.675:222):op=offset old=1850302393317 new=3190241577926
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[48850.604008] audit: type=1333 audit(1569252241.675:223):op=freq old=-2436281764244 new=-2413071187316
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[49926.567270] audit: type=1333 audit(1569253317.638:224):op=offset old=2453141035832 new=2372389610455
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[49926.567273] audit: type=1333 audit(1569253317.638:225):op=freq old=-2413071187316 new=-2403561671476
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This gets emitted every time ntp makes an adjustment, which
is apparently very frequent on some hosts.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Audit isn't even enabled on these machines.
# auditctl -l No rulesWhat happens when you run 'auditctl -a never,task'? That
*should*
Quoted 1 line
Quoted 1 line
Quoted 1 line
silence those messages as the
audit_ntp_log()function hasthe
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
requisite
audit_dummy_context()check.They still get emitted.
Quoted 3 lines
FWIW, this is the distro
default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system.As these machines aren't using audit, they aren't running auditd
either.
Quoted 1 line
Quoted 1 line
Essentially: nothing enables audit, but the kernel side
continues to log
Quoted 3 lines
Quoted 1 line
ntp regardless (no other audit messages seem to do this).
What does your kernel command line look like? Do you have
"audit=1"
Quoted 1 line
somewhere in there?
nope.
ro root=LABEL=/ biosdevname=0 net.ifnames=0 fsck.repair=yes systemd.gpt_auto=0 pcie_pme=nomsi ipv6.autoconf=0
erst_disablecrashkernel=128M console=tty0 console=ttyS1,57600 intel_iommu=tboot_noforceDave
Paul Moore
maintainerQuoted 2 lines · Eric Paris
Is this the thing where systemd is listening on the multicast netlink socket and causes everything to come out kmesg as well?
I don't think so, but I'm still a little confused as to why DaveJ is seeing these records, so I'll go with a weak "maybe" ;)
Quoted 65 lines · Dave Jones
Quoted 63 lines · Paul Moore
Quoted 1 line
On Mon, Sep 23, 2019 at 12:58 PM Dave Jones <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines · Paul Moore
Quoted 1 line
On Mon, Sep 23, 2019 at 11:50 AM Dave Jones <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I have some hosts that are constantly spewing audit
messages like so:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[46897.591182] audit: type=1333 audit(1569250288.663:220):op=offset old=2543677901372 new=2980866217213
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[46897.591184] audit: type=1333 audit(1569250288.663:221):op=freq old=-2443166611284 new=-2436281764244
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[48850.604005] audit: type=1333 audit(1569252241.675:222):op=offset old=1850302393317 new=3190241577926
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[48850.604008] audit: type=1333 audit(1569252241.675:223):op=freq old=-2436281764244 new=-2413071187316
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[49926.567270] audit: type=1333 audit(1569253317.638:224):op=offset old=2453141035832 new=2372389610455
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[49926.567273] audit: type=1333 audit(1569253317.638:225):op=freq old=-2413071187316 new=-2403561671476
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This gets emitted every time ntp makes an adjustment, which
is apparently very frequent on some hosts.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Audit isn't even enabled on these machines.
# auditctl -l No rulesWhat happens when you run 'auditctl -a never,task'? That
*should*
Quoted 1 line
Quoted 1 line
Quoted 1 line
silence those messages as the
audit_ntp_log()function hasthe
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
requisite
audit_dummy_context()check.They still get emitted.
Quoted 3 lines
FWIW, this is the distro
default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system.As these machines aren't using audit, they aren't running auditd
either.
Quoted 1 line
Quoted 1 line
Essentially: nothing enables audit, but the kernel side
continues to log
Quoted 3 lines
Quoted 1 line
ntp regardless (no other audit messages seem to do this).
What does your kernel command line look like? Do you have
"audit=1"
Quoted 1 line
somewhere in there?
nope.
ro root=LABEL=/ biosdevname=0 net.ifnames=0 fsck.repair=yes systemd.gpt_auto=0 pcie_pme=nomsi ipv6.autoconf=0
erst_disablecrashkernel=128M console=tty0 console=ttyS1,57600 intel_iommu=tboot_noforceDave
Signature · 2 lines
paul moore www.paul-moore.com
Quoted 2 lines · EDT Eric Paris
Is this the thing where systemd is listening on the multicast netlink socket and causes everything to come out kmesg as well?
Almost, it actually enables audit without knowing anything about the end user's wishes:
https://github.com/systemd/systemd/blob/master/src/journal/journald-audit.c#L458
-Steve
Quoted 84 lines · Dave Jones
Quoted 70 lines · Paul Moore
Quoted 1 line
On Mon, Sep 23, 2019 at 12:58 PM Dave Jones <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines · Paul Moore
Quoted 1 line
On Mon, Sep 23, 2019 at 11:50 AM Dave Jones <
Quoted 1 line · [email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
I have some hosts that are constantly spewing audit
messages like so:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[46897.591182] audit: type=1333 audit(1569250288.663:220):op=offset old=2543677901372 new=2980866217213
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[46897.591184] audit: type=1333 audit(1569250288.663:221):op=freq old=-2443166611284 new=-2436281764244
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[48850.604005] audit: type=1333 audit(1569252241.675:222):op=offset old=1850302393317 new=3190241577926
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[48850.604008] audit: type=1333 audit(1569252241.675:223):op=freq old=-2436281764244 new=-2413071187316
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[49926.567270] audit: type=1333 audit(1569253317.638:224):op=offset old=2453141035832 new=2372389610455
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[49926.567273] audit: type=1333 audit(1569253317.638:225):op=freq old=-2413071187316 new=-2403561671476
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This gets emitted every time ntp makes an adjustment, which
is apparently very frequent on some hosts.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Audit isn't even enabled on these machines.
# auditctl -l No rulesWhat happens when you run 'auditctl -a never,task'? That
*should*
Quoted 1 line
Quoted 1 line
Quoted 1 line
silence those messages as the
audit_ntp_log()function hasthe
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
requisite
audit_dummy_context()check.They still get emitted.
Quoted 3 lines
FWIW, this is the distro
default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system.As these machines aren't using audit, they aren't running auditd
either.
Quoted 1 line
Quoted 1 line
Essentially: nothing enables audit, but the kernel side
continues to log
Quoted 3 lines
Quoted 1 line
ntp regardless (no other audit messages seem to do this).
What does your kernel command line look like? Do you have
"audit=1"
Quoted 1 line
somewhere in there?
Quoted 8 lines
nope.
ro root=LABEL=/ biosdevname=0 net.ifnames=0 fsck.repair=yes systemd.gpt_auto=0 pcie_pme=nomsi ipv6.autoconf=0
erst_disablecrashkernel=128M console=tty0 console=ttyS1,57600 intel_iommu=tboot_noforceDave
--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit
Quoted 6 lines · Paul Moore
Quoted 5 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244
Odd. It appears these two above should have the same serial number and should be accompanied by a syscall record. It appears that it has no context to update to connect the two records. Is it possible it is not being called in a task context? If that were the case though, I'd expect audit_dummy_context() to return 1...
Checking audit_enabled should not be necessary but might fix the problem, but still not explain why we're getting these records.
Quoted 26 lines
Quoted 5 lines
[48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offset old=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476Quoted 7 lines
This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Audit isn't even enabled on these machines.
# auditctl -l No rules[NOTE: added linux-audit to the CC line]
There is an audit mailing list, please CC it when you have audit concerns/questions/etc.
What happens when you run 'auditctl -a never,task'? That *should* silence those messages as the
audit_ntp_log()function has the requisiteaudit_dummy_context()check. FWIW, this is the distro default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system. A more selective configuration could simply exclude theTIME_ADJNTPVALrecord (type 1333) from the records that the kernel emits.
Quoted 14 lines
We could also add a
audit_enabledcheck at the top of audit_ntp_log()/__audit_ntp_log(), but I imagine some of that depends on the various security requirements (they can be bizzare and I can't say I'm up to date on all those - Steve Grubb should be able to comment on that).Signature · 7 lines
paul moore www.paul-moore.com -- Linux-audit mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-audit
- RGB
--
Richard Guy Briggs <[email protected]>
Sr. S/W Engineer, Kernel Security, Base Operating Systems Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635
Paul Moore
maintainerQuoted 5 lines · Richard Guy Briggs
Quoted 4 lines · Paul Moore
Quoted 3 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244Odd. It appears these two above should have the same serial number and should be accompanied by a syscall record. It appears that it has no context to update to connect the two records. Is it possible it is not being called in a task context? If that were the case though, I'd expect
audit_dummy_context()to return 1...
Yeah, I'm a little confused with these messages too. As you pointed out, the different serial numbers imply that the audit_context is NULL and if the audit_context is NULL I would have expected it to fail the audit_dummy_context() check in audit_ntp_log(). I'm looking at this with tired eyes at the moment, so I'm likely missing something, but I just don't see it right now ...
What is even more confusing is that I don't see this issue on my test systems.
Quoted 2 lines
Checking
audit_enabledshould not be necessary but might fix the problem, but still not explain why we're getting these records.
I'd like to understand why this is happening before we start changing the code.
Signature · 2 lines
paul moore www.paul-moore.com
Quoted 22 lines · Paul Moore
Quoted 5 lines · Richard Guy Briggs
Quoted 4 lines · Paul Moore
Quoted 3 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244Odd. It appears these two above should have the same serial number and should be accompanied by a syscall record. It appears that it has no context to update to connect the two records. Is it possible it is not being called in a task context? If that were the case though, I'd expect
audit_dummy_context()to return 1...Yeah, I'm a little confused with these messages too. As you pointed out, the different serial numbers imply that the
audit_contextis NULL and if theaudit_contextis NULL I would have expected it to fail theaudit_dummy_context()check inaudit_ntp_log(). I'm looking at this with tired eyes at the moment, so I'm likely missing something, but I just don't see it right now ...
Quoted 6 lines
What is even more confusing is that I don't see this issue on my test systems.
Quoted 2 lines
Checking
audit_enabledshould not be necessary but might fix the problem, but still not explain why we're getting these records.I'd like to understand why this is happening before we start changing the code.
Absolutely.
This looks like a similar issue to the AUDIT_NETFILTER_CFG issue where we get a lone record unconnected to a syscall when one of the netfilter table initialization (ipv4 filter) is linked into the kernel rather than compiled as a module, so it is run in kernel context at boot rather than in user context as a module load later. This is why I ask if it is being run by a kernel thread rather than a user task, perhaps using a syscall function call internally.
Quoted 1 line
paul moore
- RGB
--
Richard Guy Briggs <[email protected]>
Sr. S/W Engineer, Kernel Security, Base Operating Systems Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635
Paul Moore
maintainerQuoted 32 lines · Richard Guy Briggs
Quoted 22 lines · Paul Moore
Quoted 5 lines · Richard Guy Briggs
Quoted 4 lines · Paul Moore
Quoted 3 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244Odd. It appears these two above should have the same serial number and should be accompanied by a syscall record. It appears that it has no context to update to connect the two records. Is it possible it is not being called in a task context? If that were the case though, I'd expect
audit_dummy_context()to return 1...Yeah, I'm a little confused with these messages too. As you pointed out, the different serial numbers imply that the
audit_contextis NULL and if theaudit_contextis NULL I would have expected it to fail theaudit_dummy_context()check inaudit_ntp_log(). I'm looking at this with tired eyes at the moment, so I'm likely missing something, but I just don't see it right now ...Quoted 6 lines
What is even more confusing is that I don't see this issue on my test systems.
Quoted 2 lines
Checking
audit_enabledshould not be necessary but might fix the problem, but still not explain why we're getting these records.I'd like to understand why this is happening before we start changing the code.
Absolutely.
Quoted 7 lines
This looks like a similar issue to the
AUDIT_NETFILTER_CFGissue where we get a lone record unconnected to a syscall when one of the netfilter table initialization (ipv4 filter) is linked into the kernel rather than compiled as a module, so it is run in kernel context at boot rather than in user context as a module load later. This is why I ask if it is being run by a kernel thread rather than a user task, perhaps using a syscall function call internally.
I don't see where in the code that could happen, but I agree that it looks like it; maybe I'm just missing a code path somewhere.
Is anyone else seeing these records? Granted my audit test systems are running chrony, not ntp, but the syscalls/behaviors should be similar and I can't seem to recreate this.
Signature · 2 lines
paul moore www.paul-moore.com
Paul Moore
maintainerQuoted 41 lines · Paul Moore
Quoted 32 lines · Richard Guy Briggs
Quoted 22 lines · Paul Moore
Quoted 5 lines · Richard Guy Briggs
Quoted 4 lines · Paul Moore
Quoted 3 lines · Dave Jones
I have some hosts that are constantly spewing audit messages like so:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244Odd. It appears these two above should have the same serial number and should be accompanied by a syscall record. It appears that it has no context to update to connect the two records. Is it possible it is not being called in a task context? If that were the case though, I'd expect
audit_dummy_context()to return 1...Yeah, I'm a little confused with these messages too. As you pointed out, the different serial numbers imply that the
audit_contextis NULL and if theaudit_contextis NULL I would have expected it to fail theaudit_dummy_context()check inaudit_ntp_log(). I'm looking at this with tired eyes at the moment, so I'm likely missing something, but I just don't see it right now ...Quoted 6 lines
What is even more confusing is that I don't see this issue on my test systems.
Quoted 2 lines
Checking
audit_enabledshould not be necessary but might fix the problem, but still not explain why we're getting these records.I'd like to understand why this is happening before we start changing the code.
Absolutely.
Quoted 7 lines
This looks like a similar issue to the
AUDIT_NETFILTER_CFGissue where we get a lone record unconnected to a syscall when one of the netfilter table initialization (ipv4 filter) is linked into the kernel rather than compiled as a module, so it is run in kernel context at boot rather than in user context as a module load later. This is why I ask if it is being run by a kernel thread rather than a user task, perhaps using a syscall function call internally.
Quoted 6 lines
I don't see where in the code that could happen, but I agree that it looks like it; maybe I'm just missing a code path somewhere.
Is anyone else seeing these records? Granted my audit test systems are running chrony, not ntp, but the syscalls/behaviors should be similar and I can't seem to recreate this.
Dave, can you provide any additional information on the systems where you are seeing this? Kernel, userspace, distro, relevant configs, etc.
Signature · 2 lines
paul moore www.paul-moore.com
Quoted 1 line · EDT Paul Moore
On Mon, Sep 23, 2019 at 11:50 AM Dave Jones <[email protected]>
Quoted 21 lines · wrote:
Quoted 12 lines
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offsetold=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476
Quoted 8 lines
This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Audit isn't even enabled on these machines.
# auditctl -l No rules
Quoted 6 lines
[NOTE: added linux-audit to the CC line]
There is an audit mailing list, please CC it when you have audit concerns/questions/etc.
What happens when you run 'auditctl -a never,task'?
Actually, "-e 0" should turn it off. There is a general problem where systemd turns on auditing just because it can. The above rule just makes audit processes inauditable, but does not affect the kernel originating events.
-Steve
Quoted 13 lines
That *should*
silence those messages as theaudit_ntp_log()function has the requisiteaudit_dummy_context()check. FWIW, this is the distro default for many (most? all?) distros; for example, check/etc/audit/audit.ruleson a stock Fedora system. A more selective configuration could simply exclude theTIME_ADJNTPVALrecord (type 1333) from the records that the kernel emits.We could also add a
audit_enabledcheck at the top of audit_ntp_log()/__audit_ntp_log(), but I imagine some of that depends on the various security requirements (they can be bizzare and I can't say I'm up to date on all those - Steve Grubb should be able to comment on that).
Paul Moore
maintainerQuoted 26 lines · Steve Grubb
Quoted 1 line · EDT Paul Moore
On Mon, Sep 23, 2019 at 11:50 AM Dave Jones <[email protected]>
Quoted 23 lines · wrote:
Quoted 12 lines
I have some hosts that are constantly spewing audit messages like so:
[46897.591182] audit: type=1333 audit(1569250288.663:220): op=offset old=2543677901372 new=2980866217213 [46897.591184] audit: type=1333 audit(1569250288.663:221): op=freq old=-2443166611284 new=-2436281764244 [48850.604005] audit: type=1333 audit(1569252241.675:222): op=offset old=1850302393317 new=3190241577926 [48850.604008] audit: type=1333 audit(1569252241.675:223): op=freq old=-2436281764244 new=-2413071187316 [49926.567270] audit: type=1333 audit(1569253317.638:224): op=offsetold=2453141035832 new=2372389610455 [49926.567273] audit: type=1333 audit(1569253317.638:225): op=freq old=-2413071187316 new=-2403561671476
Quoted 8 lines
This gets emitted every time ntp makes an adjustment, which is apparently very frequent on some hosts.
Audit isn't even enabled on these machines.
# auditctl -l No rules[NOTE: added linux-audit to the CC line]
Quoted 8 lines
Quoted 4 lines
There is an audit mailing list, please CC it when you have audit concerns/questions/etc.
What happens when you run 'auditctl -a never,task'?
Actually, "-e 0" should turn it off. There is a general problem where systemd turns on auditing just because it can. The above rule just makes audit processes inauditable, but does not affect the kernel originating events.
The 'auditctl -s' output was lost when I trimmed/replied to DaveJ's original email (sorry), but it appears that audit_enabled is already '0':
Quoted 9 lines
# auditctl -s enabled 0 failure 1 pid 0 rate_limit 0 backlog_limit 64 lost 0 backlog 0 loginuid_immutable 0 unlocked
Original post from DaveJ:
* https://lore.kernel.org/lkml/[email protected]
Signature · 2 lines
paul moore www.paul-moore.com
PATCH 1/1 audit: set context->dummy even when audit is off
In reply to Paul MooreDave Jones reported that we're finding a considerable amount of dmesg traffic from NTP time adjustments being reported through the audit subsystem. His original post is here:
https://lore.kernel.org/lkml/[email protected]/
The confusing part is that we're seeing this on machines that don't have audit on. The NTP code uses audit_dummy_context() to decide if it should log things:
static inline void audit_ntp_log(const struct audit_ntp_data *ad)
{
if (!audit_dummy_context())
__audit_ntp_log(ad);
}I confirmed with perf probes that:
context->dummy = 0
audit_n_rules = 0
audit_enabled = 0
audit_ever_enabled = 1 // seems to be from journald
The box boots, journald turns audit on, some time later our configuration management runs around and turns audit off. This journald feature is discussed here: https://github.com/systemd/systemd/issues/959
Quoted 1 line
From what I can tell,
audit_syscall_entryis responsible for setting
context->dummy, but we never get down to the test for audit_n_rules:
__audit_syscall_entry(int major, unsigned long a1, unsigned long a2,
unsigned long a3, unsigned long a4)
{
struct audit_context *context = audit_context();
enum audit_state state;
if (!audit_enabled || !context)
return;
^^^^^^^^^^^^^^^^^^ --- we bail here
[ ... ]
context->dummy = !audit_n_rules;This leaves context->dummy at 0, which appears to be the original value from kzalloc().
If you've gotten this far, you've read everything I know about the audit code. With that said, my preference is to make a single source of truth for decisions about logging. This commit changes __audit_syscall_entry() to set context->dummy when audit is off.
--- kernel/auditsc.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
kernel/auditsc.c +12 -1
@@ -1631,8 +1631,19 @@ void __audit_syscall_entry(int major, unsigned long a1, unsigned long a2,struct audit_context *context =audit_context ();enum audit_state state; -if (!audit_enabled || !context) +if (!context) +return ; + +if (!audit_enabled) { +/* +* ntp clock adjustments and a few other places check for +* a dummy context without checking to see if audit +* is enabled. Make sure we set context->dummy when audit +* is off, otherwise they will try to log things. +*/ + context->dummy =1 ;return ; + }BUG_ON (context->in_syscall || context->name_count);
Signature · 1 line
2.17.1
Paul Moore
maintainerRe: [PATCH] audit: set context->dummy even when audit is off
Quoted 52 lines · Chris Mason
Dave Jones reported that we're finding a considerable amount of dmesg traffic from NTP time adjustments being reported through the audit subsystem. His original post is here:
https://lore.kernel.org/lkml/[email protected]/
The confusing part is that we're seeing this on machines that don't have audit on. The NTP code uses
audit_dummy_context()to decide if it should log things:static inline void audit_ntp_log(const struct audit_ntp_data *ad) { if (!audit_dummy_context()) __audit_ntp_log(ad); }I confirmed with perf probes that:
context->dummy= 0
audit_n_rules= 0
audit_enabled= 0
audit_ever_enabled= 1 // seems to be from journaldThe box boots, journald turns audit on, some time later our configuration management runs around and turns audit off. This journald feature is discussed here: https://github.com/systemd/systemd/issues/959
From what I can tell,
audit_syscall_entryis responsible for settingcontext->dummy, but we never get down to the test foraudit_n_rules:__audit_syscall_entry(int major, unsigned long a1, unsigned long a2,
unsigned long a3, unsigned long a4){ struct audit_context *context = audit_context(); enum audit_state state; if (!audit_enabled || !context) return; ^^^^^^^^^^^^^^^^^^ --- we bail here [ ... ] context->dummy = !audit_n_rules;This leaves
context->dummyat 0, which appears to be the original value fromkzalloc().If you've gotten this far, you've read everything I know about the audit code. With that said, my preference is to make a single source of truth for decisions about logging. This commit changes
__audit_syscall_entry()to setcontext->dummywhen audit is off.
Quoted 5 lines
Reported-by Dave Jones <[email protected]>Signed-off-by Chris Mason <[email protected]>--- kernel/auditsc.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Hi Chris,
This is a rather hasty email as I'm at a conference right now, but I wanted to convey that I'm not opposed to making sure that the NTP records obey the audit configuration (that was the original intent after all), I think it is just that we are all a little confused as to why you are seeing the NTP records *and*only* the NTP records.
It's been a while, but I thought we suggested Dave try running 'auditctl -a never,task' to see if that would solve his problem and I believe his answer was no, which confused me a bit as the audit_filter_task() call in audit_alloc() should see that rule and return a state of AUDIT_DISABLED which not only prevents audit_alloc() from allocating an audit_context (and remember if the audit_context is NULL then audit_dummy_context() returns true), but it also clears the TIF_SYSCALL_AUDIT flag (which I'm guessing you also want).
Can you confirm the results of 'auditctl -a never,task' on your systems?
Signature · 2 lines
paul moore www.paul-moore.com
Re: [PATCH] audit: set context->dummy even when audit is off
Quoted 9 lines · Paul Moore
On Thu, Oct 31, 2019 at 12:40 PM Chris Mason <[email protected]> wrote:
[ ... ]
Hi Chris,This is a rather hasty email as I'm at a conference right now, but I wanted to convey that I'm not opposed to making sure that the NTP records obey the audit configuration (that was the original intent after all), I think it is just that we are all a little confused as to why you are seeing the NTP records *and*only* the NTP records.
This part is harder to nail down because there's a window during boot where journald has enabled audit but chef hasn't yet run in and turned it off, so we get a lot of logs early and then mostly ntp after that.
I feel like the answer is that most of the places that actually log audit records are also checking audit_enabled. Poking a bit with cscope, we're not using most of the places that rely only on audit_dummy_context().
I grabbed the last week or so of audit: lines from netconsole, and most of them (73%) were type 1130 from early in the boot. These are the ones turned off when chef runs. Another big chunk were the one time audit initialized message from boot, mostly reflecting our rollout of the new kernel. After that it was ntp and couple of random things like fanotify.
Quoted 10 lines
It's been a while, but I thought we suggested Dave try running 'auditctl -a never,task' to see if that would solve his problem and I believe his answer was no, which confused me a bit as the
audit_filter_task()call inaudit_alloc()should see that rule and return a state ofAUDIT_DISABLEDwhich not only preventsaudit_alloc()from allocating anaudit_context(and remember if theaudit_contextis NULL thenaudit_dummy_context()returns true), but it also clears theTIF_SYSCALL_AUDITflag (which I'm guessing you also want).
Thanks for the reminder on this part, I meant to test it. Yes, auditctl -a never,task does stop the messages, even without my patch applied.
-chris
Re: [PATCH] audit: set context->dummy even when audit is off
Quoted 11 lines · EDT Chris Mason
Quoted 9 lines · Paul Moore
On Thu, Oct 31, 2019 at 12:40 PM Chris Mason <[email protected]> wrote:
[ ... ]
Hi Chris,This is a rather hasty email as I'm at a conference right now, but I wanted to convey that I'm not opposed to making sure that the NTP records obey the audit configuration (that was the original intent after all), I think it is just that we are all a little confused as to why you are seeing the NTP records *and*only* the NTP records.
Quoted 3 lines
This part is harder to nail down because there's a window during boot where journald has enabled audit but chef hasn't yet run in and turned it off, so we get a lot of logs early and then mostly ntp after that.
This is the root of the problem. Journald should never turn on audit since it has no idea if auditd even has rules to load. What if the end user does not want auditing? By blindly enabling audit without knowing if its wanted, it causes a system performance hit even with no rules loaded. It would be best if journald leaves audit alone. If it wants to listen on the multicast socket, so be it. It should just listen and not try to alter the system.
Back to ntp, it sounds like the ntp record needs to check for audit_enabled rather than the dummy context.
-Steve
Re: [PATCH] audit: set context->dummy even when audit is off
Quoted 6 lines · Steve Grubb
This is the root of the problem. Journald should never turn on audit since it has no idea if auditd even has rules to load. What if the end user does not want auditing? By blindly enabling audit without knowing if its wanted, it causes a system performance hit even with no rules loaded. It would be best if journald leaves audit alone. If it wants to listen on the multicast socket, so be it. It should just listen and not try to alter the system.
+1 for me, except I would also question why it would even listen, as to me it seems that implies storage.
If that's true, I would want to be able to disable it as I do not want audit events stored elsewhere as well.
Thx,
LCB
Signature · 2 lines
Lenny Bruzenak MagitekLTD
Re: [PATCH] audit: set context->dummy even when audit is off
Quoted 9 lines · EDT Lenny Bruzenak
Quoted 7 lines · Steve Grubb
This is the root of the problem. Journald should never turn on audit since it has no idea if auditd even has rules to load. What if the end user does not want auditing? By blindly enabling audit without knowing if its wanted, it causes a system performance hit even with no rules loaded. It would be best if journald leaves audit alone. If it wants to listen on the multicast socket, so be it. It should just listen and not try to alter the system.
Quoted 5 lines
+1 for me, except I would also question why it would even listen, as to me it seems that implies storage.
If that's true, I would want to be able to disable it as I do not want audit events stored elsewhere as well.
It is true. You get 2 copies, one in the journal and it also relays one to rsyslog. This should fix it:
systemctl mask systemd-journald-audit.socket
-Steve
Re: [PATCH] audit: set context->dummy even when audit is off
Quoted 11 lines · Steve Grubb
Quoted 5 lines
+1 for me, except I would also question why it would even listen, as to me it seems that implies storage.
If that's true, I would want to be able to disable it as I do not want audit events stored elsewhere as well.
It is true. You get 2 copies, one in the journal and it also relays one to rsyslog. This should fix it:
systemctl mask systemd-journald-audit.socket
-Steve
Gotcha; thanks Steve.
LCB
Signature · 2 lines
Lenny Bruzenak MagitekLTD
Re: [PATCH] audit: set context->dummy even when audit is off
In reply to Steve GrubbQuoted 30 lines · Steve Grubb
Quoted 12 lines · EDT Chris Mason
Quoted 10 lines · Paul Moore
On Thu, Oct 31, 2019 at 12:40 PM Chris Mason <[email protected]> wrote:
[ ... ]
Hi Chris,This is a rather hasty email as I'm at a conference right now, but I wanted to convey that I'm not opposed to making sure that the NTP records obey the audit configuration (that was the original intent after all), I think it is just that we are all a little confused as to
why you are seeing the NTP records *and*only* the NTP records.Quoted 4 lines
This part is harder to nail down because there's a window during boot where journald has enabled audit but chef hasn't yet run in and turned
it off, so we get a lot of logs early and then mostly ntp after that.This is the root of the problem. Journald should never turn on audit since it
has no idea if auditd even has rules to load. What if the end user does not
want auditing? By blindly enabling audit without knowing if its wanted, it
causes a system performance hit even with no rules loaded. It would be best
if journald leaves audit alone. If it wants to listen on the multicast socket, so be it. It should just listen and not try to alter the system.
Quoted 3 lines
Back to ntp, it sounds like the ntp record needs to check for
audit_enabled
rather than the dummy context.
I'm not against sprinkling more audit_enabled checks, but we'd have to change audit_inode() and a bunch of the other callers of audit_dummy_context() as well.
-chris
Paul Moore
maintainerRe: [PATCH] audit: set context->dummy even when audit is off
In reply to Chris MasonQuoted 10 lines · Chris Mason
Quoted 8 lines · Paul Moore
It's been a while, but I thought we suggested Dave try running 'auditctl -a never,task' to see if that would solve his problem and I believe his answer was no, which confused me a bit as the
audit_filter_task()call inaudit_alloc()should see that rule and return a state ofAUDIT_DISABLEDwhich not only preventsaudit_alloc()from allocating anaudit_context(and remember if theaudit_contextis NULL thenaudit_dummy_context()returns true), but it also clears theTIF_SYSCALL_AUDITflag (which I'm guessing you also want).
Quoted 2 lines
Thanks for the reminder on this part, I meant to test it. Yes, auditctl -a never,task does stop the messages, even without my patch applied.
I'm glad to hear that worked, I was going to be *very* confused if you came back and said you were still seeing NTP records.
I would suggest that regardless of what happens with audit_enabled you likely want to keep this audit rule as part of your boot configuration, not only does it squelch the audit records, but it should improve performance as well (at the cost of no syscall auditing). A number of Linux distros have this as their default at boot.
Signature · 2 lines
paul moore www.paul-moore.com
Re: [PATCH] audit: set context->dummy even when audit is off
Quoted 22 lines · Paul Moore
Quoted 14 lines · Chris Mason
Quoted 4 lines · Paul Moore
It's been a while, but I thought we suggested Dave try running 'auditctl -a never,task' to see if that would solve his problem and I
believe his answer was no, which confused me a bit as theQuoted 8 lines
audit_filter_task()call inaudit_alloc()should see that rule and return a state ofAUDIT_DISABLEDwhich not only preventsaudit_alloc()
from allocating anaudit_context(and remember if theaudit_contextis
NULL thenaudit_dummy_context()returns true), but it also clears theTIF_SYSCALL_AUDITflag (which I'm guessing you also want).Quoted 3 lines
Thanks for the reminder on this part, I meant to test it. Yes, auditctl
-a never,task does stop the messages, even without my patch applied.I'm glad to hear that worked, I was going to be *very* confused if you came back and said you were still seeing NTP records.
Quoted 7 lines
I would suggest that regardless of what happens with
audit_enabledyou likely want to keep this audit rule as part of your boot configuration, not only does it squelch the audit records, but it should improve performance as well (at the cost of no syscall auditing). A number of Linux distros have this as their default at boot.
Definitely, we'll be testing auditctl -a never,task internally. Before we went down that path I wanted to fully understand what was going on, but I think all the big questions have been answered at this point.
I'm happy to try variations on my patch, but if you want to include it, please do remember that I've really only tested it with auditing off.
-chris
Paul Moore
maintainerRe: [PATCH] audit: set context->dummy even when audit is off
Quoted 32 lines · Chris Mason
Quoted 22 lines · Paul Moore
Quoted 14 lines · Chris Mason
Quoted 4 lines · Paul Moore
It's been a while, but I thought we suggested Dave try running 'auditctl -a never,task' to see if that would solve his problem and I
believe his answer was no, which confused me a bit as theQuoted 8 lines
audit_filter_task()call inaudit_alloc()should see that rule and return a state ofAUDIT_DISABLEDwhich not only preventsaudit_alloc()
from allocating anaudit_context(and remember if theaudit_contextis
NULL thenaudit_dummy_context()returns true), but it also clears theTIF_SYSCALL_AUDITflag (which I'm guessing you also want).Quoted 3 lines
Thanks for the reminder on this part, I meant to test it. Yes, auditctl
-a never,task does stop the messages, even without my patch applied.I'm glad to hear that worked, I was going to be *very* confused if you came back and said you were still seeing NTP records.
Quoted 7 lines
I would suggest that regardless of what happens with
audit_enabledyou likely want to keep this audit rule as part of your boot configuration, not only does it squelch the audit records, but it should improve performance as well (at the cost of no syscall auditing). A number of Linux distros have this as their default at boot.
Quoted 3 lines
Definitely, we'll be testing auditctl -a never,task internally. Before we went down that path I wanted to fully understand what was going on, but I think all the big questions have been answered at this point.
Yes, that is why we didn't do anything earlier with Dave's reports; we couldn't reconcile the results with the code, and the lack of other similar problem reports made me suspicious. As you said, we understand things a bit better now.
Quoted 2 lines
I'm happy to try variations on my patch, but if you want to include it, please do remember that I've really only tested it with auditing off.
Understood. FWIW, I'm not overly in love with the approach in the patch you posted, but I haven't looked too seriously into alternatives thus far. I expect with most everyone running with the "never" audit rule installed this solves this just fine for the time being.
Signature · 2 lines
paul moore www.paul-moore.com