ratelimit printk messages from the audit system
Eric Paris
maintainerPATCH 1/1 ratelimit printk messages from the audit system
Some printk messages from the audit system can become excessive. This patch ratelimits those messages. It was found that messages, such as the audit backlog lost printk message could flood the logs to the point that a machine could take an nmi watchdog hit or otherwise become unresponsive.
--- kernel/audit.c | 28 ++++++++++++++++++---------- 1 files changed, 18 insertions(+), 10 deletions(-)
kernel/audit.c +18 -10
@@ -163,7 +163,8 @@ void audit_panic(const char *message)case AUDIT_FAIL_SILENT :break ;case AUDIT_FAIL_PRINTK : -printk (KERN_ERR "audit: %s\n" , message); +if (printk_ratelimit ()) +printk (KERN_ERR "audit: %s\n" , message);break ;case AUDIT_FAIL_PANIC :panic ("audit: %s\n" , message);
@@ -231,11 +232,13 @@ void audit_log_lost(const char *message) }if (print) { -printk (KERN_WARNING -"audit: audit_lost=%d audit_rate_limit=%d audit_backlog_limit=%d\n" , -atomic_read (&audit_lost), - audit_rate_limit, - audit_backlog_limit); +if (printk_ratelimit ()) +printk (KERN_WARNING +"audit: audit_lost=%d audit_rate_limit=%d " +"audit_backlog_limit=%d\n" , +atomic_read (&audit_lost), + audit_rate_limit, + audit_backlog_limit);audit_panic (message); } }
@@ -405,7 +408,11 @@ static int kauditd_thread(void *dummy) audit_pid =0 ; } }else { -printk (KERN_NOTICE "%s\n" , skb->data +NLMSG_SPACE (0 )); +if (printk_ratelimit ()) +printk (KERN_NOTICE "%s\n" , skb->data + +NLMSG_SPACE (0 )); +else +audit_log_lost ("printk limit exceeded\n" );kfree_skb (skb); } }else {
@@ -1164,7 +1171,7 @@ struct audit_buffer *audit_log_start(struct audit_context *ctx, gfp_t gfp_mask,remove_wait_queue (&audit_backlog_wait, &wait);continue ; } -if (audit_rate_check ()) +if (audit_rate_check () &&printk_ratelimit ()) printk(KERN_WARNING "audit: audit_backlog=%d > " "audit_backlog_limit=%d\n",
@@ -1433,9 +1440,10 @@ void audit_log_end(struct audit_buffer *ab)skb_queue_tail (&audit_skb_queue, ab->skb); ab->skb =NULL ;wake_up_interruptible (&kauditd_wait); - }else { + }else if (printk_ratelimit ())printk (KERN_NOTICE "%s\n" , ab->skb->data +NLMSG_SPACE (0 )); - } +else +audit_log_lost ("printk limit exceeded\n" ); }audit_buffer_free (ab); }
Quoted 28 lines · Eric Paris
Some printk messages from the audit system can become excessive. This patch ratelimits those messages. It was found that messages, such as the audit backlog lost printk message could flood the logs to the point that a machine could take an nmi watchdog hit or otherwise become unresponsive.
Signed-off-by Eric Paris <[email protected]>--- kernel/audit.c | 28 ++++++++++++++++++---------- 1 files changed, 18 insertions(+), 10 deletions(-)kernel/audit.c +2
-1@@ -163,7 +163,8 @@ void audit_panic(const char *message)case AUDIT_FAIL_SILENT :break ;case AUDIT_FAIL_PRINTK : -printk (KERN_ERR "audit: %s\n" , message); +if (printk_ratelimit ()) +printk (KERN_ERR "audit: %s\n" , message);break ;case AUDIT_FAIL_PANIC :panic ("audit: %s\n" , message);@@ -231,11 +232,13 @@ void audit_log_lost(const char *message) }
kernel/audit.c void audit_log_lost(const char *message) · line 237
Quoted 10 lines
kernel/audit.c +3
-5if (print) { -printk (KERN_WARNING -"audit: audit_lost=%d audit_rate_limit=%d audit_backlog_limit=%d\n" , -atomic_read (&audit_lost), - audit_rate_limit, - audit_backlog_limit); +if (printk_ratelimit ()) +printk (KERN_WARNING + "audit: audit_lost=%d audit_rate_limit=%d "
This is unrelated to your patch but I think it would be nice if audit_lost represented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?
Quoted 8 lines
kernel/audit.c +4
-0+"audit_backlog_limit=%d\n" , +atomic_read (&audit_lost), + audit_rate_limit, + audit_backlog_limit);audit_panic (message); } }
kernel/audit.c static int kauditd_thread(void *dummy) · line 415
Quoted 9 lines
kernel/audit.c +5
-1@@ -405,7 +408,11 @@ static int kauditd_thread(void *dummy) audit_pid =0 ; } }else { -printk (KERN_NOTICE "%s\n" , skb->data +NLMSG_SPACE (0 )); +if (printk_ratelimit ()) +printk (KERN_NOTICE "%s\n" , skb->data + +NLMSG_SPACE (0 )); +else +audit_log_lost ("printk limit exceeded\n" );
If you call audit_log_lost when the printk limit is exceeded, but then audit_log_lost also checks the printk limit, will this message ever come out? Does it make sense to print a message saying we couldn't print a message?
Quoted 13 lines
kernel/audit.c +1
-1kfree_skb (skb); } }else {@@ -1164,7 +1171,7 @@ struct audit_buffer *audit_log_start(struct audit_context *ctx, gfp_t gfp_mask,remove_wait_queue (&audit_backlog_wait, &wait);continue ; } -if (audit_rate_check ()) +if (audit_rate_check () &&printk_ratelimit ()) printk(KERN_WARNING "audit: audit_backlog=%d > " "audit_backlog_limit=%d\n",
kernel/audit.c void audit_log_end(struct audit_buffer *ab) · line 1446
Quoted 9 lines
kernel/audit.c +3
-2@@ -1433,9 +1440,10 @@ void audit_log_end(struct audit_buffer *ab)skb_queue_tail (&audit_skb_queue, ab->skb); ab->skb =NULL ;wake_up_interruptible (&kauditd_wait); - }else { + }else if (printk_ratelimit ())printk (KERN_NOTICE "%s\n" , ab->skb->data +NLMSG_SPACE (0 )); - } +else +audit_log_lost ("printk limit exceeded\n" );
Same question here.
I wonder if it would be better to reduce the generation of the messages, rather than just their output. For example, once we're losing records, should we just flush the queue, issue one message, and then keep going? Or perhaps issue one message, shut off incoming so we don't accept new records until the backlog goes to zero, then start up again?
Quoted 9 lines
kernel/audit.c +0
-1}audit_buffer_free (ab); } --Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit
Eric Paris
maintainerQuoted 30 lines · Linda Knippers
Quoted 29 lines · Eric Paris
Some printk messages from the audit system can become excessive. This patch ratelimits those messages. It was found that messages, such as the audit backlog lost printk message could flood the logs to the point that a machine could take an nmi watchdog hit or otherwise become unresponsive.
Signed-off-by Eric Paris <[email protected]>--- kernel/audit.c | 28 ++++++++++++++++++---------- 1 files changed, 18 insertions(+), 10 deletions(-)kernel/audit.c +2
-1@@ -163,7 +163,8 @@ void audit_panic(const char *message)case AUDIT_FAIL_SILENT :break ;case AUDIT_FAIL_PRINTK : -printk (KERN_ERR "audit: %s\n" , message); +if (printk_ratelimit ()) +printk (KERN_ERR "audit: %s\n" , message);break ;case AUDIT_FAIL_PANIC :panic ("audit: %s\n" , message);@@ -231,11 +232,13 @@ void audit_log_lost(const char *message) }
Quoted 15 lines
kernel/audit.c void audit_log_lost(const char *message) · line 237
Quoted 9 lines
kernel/audit.c +3
-5if (print) { -printk (KERN_WARNING -"audit: audit_lost=%d audit_rate_limit=%d audit_backlog_limit=%d\n" , -atomic_read (&audit_lost), - audit_rate_limit, - audit_backlog_limit); +if (printk_ratelimit ()) +printk (KERN_WARNING + "audit: audit_lost=%d audit_rate_limit=%d "This is unrelated to your patch but I think it would be nice if
audit_lostrepresented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?
Shouldn't be too much overhead, we are already on a slow/unlikely path. What's the benefit though? Just don't want to have to do a subtraction? If we are dropping the 'we lost some messages' message 0'ing the counter at that time would be a bad idea, certainly not unsolvable, but I don't see what it buys us.
Quoted 15 lines
Quoted 15 lines
kernel/audit.c +6
-1+"audit_backlog_limit=%d\n" , +atomic_read (&audit_lost), + audit_rate_limit, + audit_backlog_limit);audit_panic (message); } }@@ -405,7 +408,11 @@ static int kauditd_thread(void *dummy) audit_pid =0 ; } }else { -printk (KERN_NOTICE "%s\n" , skb->data +NLMSG_SPACE (0 )); +if (printk_ratelimit ()) + printk(KERN_NOTICE "%s\n", skb->data +
Quoted 8 lines
kernel/audit.c static int kauditd_thread(void *dummy) · line 415
Quoted 3 lines
kernel/audit.c +3
-0+ NLMSG_SPACE(0)); + else + audit_log_lost("printk limit exceeded\n");If you call
audit_log_lostwhen the printk limit is exceeded, but thenaudit_log_lostalso checks the printk limit, will this message ever come out? Does it make sense to print a message saying we couldn't print a message?
No it won't come out of audit_log_lost() through printk either, but what it does do is call audit_panic() and we get the lost message accounting.
Quoted 25 lines
Quoted 13 lines
kernel/audit.c +1
-1kfree_skb (skb); } }else {@@ -1164,7 +1171,7 @@ struct audit_buffer *audit_log_start(struct audit_context *ctx, gfp_t gfp_mask,remove_wait_queue (&audit_backlog_wait, &wait);continue ; } -if (audit_rate_check ()) +if (audit_rate_check () &&printk_ratelimit ()) printk(KERN_WARNING "audit: audit_backlog=%d > " "audit_backlog_limit=%d\n",kernel/audit.c void audit_log_end(struct audit_buffer *ab) · line 1446
Quoted 9 lines
kernel/audit.c +3
-2@@ -1433,9 +1440,10 @@ void audit_log_end(struct audit_buffer *ab)skb_queue_tail (&audit_skb_queue, ab->skb); ab->skb =NULL ;wake_up_interruptible (&kauditd_wait); - }else { + }else if (printk_ratelimit ())printk (KERN_NOTICE "%s\n" , ab->skb->data +NLMSG_SPACE (0 )); - } +else +audit_log_lost ("printk limit exceeded\n" );Same question here.
Quoted 3 lines
I wonder if it would be better to reduce the generation of the messages, rather than just their output. For example, once we're losing records, should we just flush the queue, issue one message, and then keep going?
I'd be scared it'd just fill up too quickly again...
Quoted 2 lines
Or perhaps issue one message, shut off incoming so we don't accept new records until the backlog goes to zero, then start up again?
Well that's sorta what we do now, we throw stuff on the floor until it gets low, maybe not to 0, i don't remember. I'll take a look.
I'll think about it, but really, as long as we are generating audit events there isn't a great way to solve this problem other than throwing stuff on the floor.
At this point I think this patch is good, but I'll look at how we handle lost messages a little more....
Quoted 46 lines · Eric Paris
Quoted 30 lines · Linda Knippers
Quoted 29 lines · Eric Paris
Some printk messages from the audit system can become excessive. This patch ratelimits those messages. It was found that messages, such as the audit backlog lost printk message could flood the logs to the point that a machine could take an nmi watchdog hit or otherwise become unresponsive.
Signed-off-by Eric Paris <[email protected]>--- kernel/audit.c | 28 ++++++++++++++++++---------- 1 files changed, 18 insertions(+), 10 deletions(-)kernel/audit.c +2
-1@@ -163,7 +163,8 @@ void audit_panic(const char *message)case AUDIT_FAIL_SILENT :break ;case AUDIT_FAIL_PRINTK : -printk (KERN_ERR "audit: %s\n" , message); +if (printk_ratelimit ()) +printk (KERN_ERR "audit: %s\n" , message);break ;case AUDIT_FAIL_PANIC :panic ("audit: %s\n" , message);@@ -231,11 +232,13 @@ void audit_log_lost(const char *message) }Quoted 14 lines
kernel/audit.c void audit_log_lost(const char *message) · line 237
Quoted 9 lines
kernel/audit.c +3
-5if (print) { -printk (KERN_WARNING -"audit: audit_lost=%d audit_rate_limit=%d audit_backlog_limit=%d\n" , -atomic_read (&audit_lost), - audit_rate_limit, - audit_backlog_limit); +if (printk_ratelimit ()) +printk (KERN_WARNING + "audit: audit_lost=%d audit_rate_limit=%d "This is unrelated to your patch but I think it would be nice if
audit_lostrepresented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?
Quoted 2 lines
Shouldn't be too much overhead, we are already on a slow/unlikely path. What's the benefit though? Just don't want to have to do a subtraction?
Well that, plus if the system is up for a long time (which we hope) and the message is infrequent (which we also hope), then it could take me a while to find the previous message in order to do the subtraction.
Quoted 3 lines
If we are dropping the 'we lost some messages' message 0'ing the counter at that time would be a bad idea, certainly not unsolvable, but I don't see what it buys us.
I wouldn't want to lose the message, just make it more useful. And if we zero it we don't have to worry about it wrapping. As it is now, its really just the count since the last time it wrapped.
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
kernel/audit.c +5
-1+"audit_backlog_limit=%d\n" , +atomic_read (&audit_lost), + audit_rate_limit, + audit_backlog_limit);audit_panic (message); } }@@ -405,7 +408,11 @@ static int kauditd_thread(void *dummy) audit_pid =0 ; } }else { -printk (KERN_NOTICE "%s\n" , skb->data +NLMSG_SPACE (0 )); +if (printk_ratelimit ())Quoted 8 lines
kernel/audit.c static int kauditd_thread(void *dummy) · line 415
Quoted 4 lines
kernel/audit.c +4
-0+printk (KERN_NOTICE "%s\n" , skb->data + +NLMSG_SPACE (0 )); + else + audit_log_lost("printk limit exceeded\n");If you call
audit_log_lostwhen the printk limit is exceeded, but thenaudit_log_lostalso checks the printk limit, will this message ever come out? Does it make sense to print a message saying we couldn't print a message?
Quoted 2 lines
No it won't come out of
audit_log_lost()through printk either, but what it does do is callaudit_panic()and we get the lost message accounting.
But audit_panic also does a rate limit check, depending on the setting of audit_failure?
Quoted 30 lines
Quoted 26 lines
Quoted 15 lines
kernel/audit.c +1
-1kfree_skb (skb); } }else {@@ -1164,7 +1171,7 @@ struct audit_buffer *audit_log_start(struct audit_context *ctx, gfp_t gfp_mask,remove_wait_queue (&audit_backlog_wait, &wait);continue ; } -if (audit_rate_check ()) +if (audit_rate_check () &&printk_ratelimit ()) printk(KERN_WARNING "audit: audit_backlog=%d > " "audit_backlog_limit=%d\n",kernel/audit.c void audit_log_end(struct audit_buffer *ab) · line 1446
Quoted 9 lines
kernel/audit.c +3
-2@@ -1433,9 +1440,10 @@ void audit_log_end(struct audit_buffer *ab)skb_queue_tail (&audit_skb_queue, ab->skb); ab->skb =NULL ;wake_up_interruptible (&kauditd_wait); - }else { + }else if (printk_ratelimit ())printk (KERN_NOTICE "%s\n" , ab->skb->data +NLMSG_SPACE (0 )); - } +else +audit_log_lost ("printk limit exceeded\n" );Same question here.
Quoted 3 lines
I wonder if it would be better to reduce the generation of the messages, rather than just their output. For example, once we're losing records, should we just flush the queue, issue one message, and then keep going?
Quoted 7 lines
I'd be scared it'd just fill up too quickly again...
Quoted 2 lines
Or perhaps issue one message, shut off incoming so we don't accept new records until the backlog goes to zero, then start up again?
Well that's sorta what we do now, we throw stuff on the floor until it gets low, maybe not to 0, i don't remember. I'll take a look.
I think it will wait a short while for there to be room in the queue before failing, but it doesn't wait for the queue to really drain. If there's one slot left, it takes it, so even if the input rate fairly closely matches the output rate, we essentially have no buffering.
Quoted 4 lines
I'll think about it, but really, as long as we are generating audit events there isn't a great way to solve this problem other than throwing stuff on the floor.
I'm actually ok with throwing stuff on the floor if that's how the audit system is configured. In fact I'm suggesting throwing more stuff on the floor until some low watermark is hit so we can actually get out of the backlog condition. Sure, it might re-occur again, but the idea is to not have an audit message rate problem immediately turn into a printk rate problem to the point that we don't actually know what we're losing anymore.
Quoted 3 lines
At this point I think this patch is good, but I'll look at how we handle lost messages a little more....
Ok, thanks. I wish I had an alternate patch to propose.
-- ljk
Quoted 17 lines · Linda Knippers
Quoted 10 lines · Eric Paris
Quoted 5 lines · Linda Knippers
This is unrelated to your patch but I think it would be nice if
audit_lostrepresented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?Shouldn't be too much overhead, we are already on a slow/unlikely path. What's the benefit though? Just don't want to have to do a subtraction?
Well that, plus if the system is up for a long time (which we hope) and the message is infrequent (which we also hope), then it could take me a while to find the previous message in order to do the subtraction.
Quoted 7 lines
Quoted 3 lines
If we are dropping the 'we lost some messages' message 0'ing the counter at that time would be a bad idea, certainly not unsolvable, but I don't see what it buys us.
I wouldn't want to lose the message, just make it more useful. And if we zero it we don't have to worry about it wrapping. As it is now, its really just the count since the last time it wrapped.
I like Linda's idea of zero'ing the lost message counter once we are able to start sending messages again for all the reasons listed above. I haven't looked at the audit message sending code, but we are only talking about adding an extra conditional in the common case and in the worst case a conditional and an assignment. Granted they are atomic ops, but everyone keeps telling me that atomic ops are pretty quick on almost all of the platforms that Linux supports ...
Signature · 2 lines
paul moore linux security @ hp
Eric Paris
maintainerQuoted 26 lines · Paul Moore
Quoted 17 lines · Linda Knippers
Quoted 10 lines · Eric Paris
Quoted 5 lines · Linda Knippers
This is unrelated to your patch but I think it would be nice if
audit_lostrepresented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?Shouldn't be too much overhead, we are already on a slow/unlikely path. What's the benefit though? Just don't want to have to do a subtraction?
Well that, plus if the system is up for a long time (which we hope) and the message is infrequent (which we also hope), then it could take me a while to find the previous message in order to do the subtraction.
Quoted 7 lines
Quoted 3 lines
If we are dropping the 'we lost some messages' message 0'ing the counter at that time would be a bad idea, certainly not unsolvable, but I don't see what it buys us.
I wouldn't want to lose the message, just make it more useful. And if we zero it we don't have to worry about it wrapping. As it is now, its really just the count since the last time it wrapped.
Quoted 7 lines
I like Linda's idea of zero'ing the lost message counter once we are able to start sending messages again for all the reasons listed above. I haven't looked at the audit message sending code, but we are only talking about adding an extra conditional in the common case and in the worst case a conditional and an assignment. Granted they are atomic ops, but everyone keeps telling me that atomic ops are pretty quick on almost all of the platforms that Linux supports ...
Delivery of audit lost messages is through printk/syslog. Assuming we can assure it gets out of printk when we reset the counter we can't assure that it made it to syslog. That means we could lose that message and have no record of it at all, nor any chance that in the future it would get recorded that it was lost either.
I wouldn't NAK such a patch, but at the same time don't anyone expect me
to write it :)
-Eric
Quoted 37 lines · Eric Paris
Quoted 27 lines · Paul Moore
Quoted 17 lines · Linda Knippers
Quoted 10 lines · Eric Paris
Quoted 5 lines · Linda Knippers
This is unrelated to your patch but I think it would be nice if
audit_lostrepresented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?Shouldn't be too much overhead, we are already on a slow/unlikely path. What's the benefit though? Just don't want to have to do a subtraction?
Well that, plus if the system is up for a long time (which we hope) and the message is infrequent (which we also hope), then it could take me a while to find the previous message in order to do the subtraction.
Quoted 8 lines
Quoted 3 lines
If we are dropping the 'we lost some messages' message 0'ing the counter at that time would be a bad idea, certainly not unsolvable, but I don't see what it buys us.
I wouldn't want to lose the message, just make it more useful. And if we zero it we don't have to worry about it wrapping. As it is now, its really just the count since the last time it wrapped.
Quoted 8 lines
I like Linda's idea of zero'ing the lost message counter once we are able to start sending messages again for all the reasons listed above. I haven't looked at the audit message sending code, but we are only talking about adding an extra conditional in the common case and in the worst case a conditional and an assignment. Granted they are atomic ops, but everyone keeps telling me that atomic ops are pretty quick on almost all of the platforms that Linux supports ...
Quoted 5 lines
Delivery of audit lost messages is through printk/syslog. Assuming we can assure it gets out of printk when we reset the counter we can't assure that it made it to syslog. That means we could lose that message and have no record of it at all, nor any chance that in the future it would get recorded that it was lost either.
That sort of begs the question - why do we even bother printing the audit record lost message?
:)
Quoted 2 lines
I wouldn't NAK such a patch, but at the same time don't anyone expect
me to write it :)
<mumbling>
... everytime I open my mouth I end up with more work ... </mumbling>
Signature · 2 lines
paul moore linux security @ hp
Eric Paris
maintainerQuoted 44 lines · Paul Moore
Quoted 37 lines · Eric Paris
Quoted 27 lines · Paul Moore
Quoted 17 lines · Linda Knippers
Quoted 10 lines · Eric Paris
Quoted 5 lines · Linda Knippers
This is unrelated to your patch but I think it would be nice if
audit_lostrepresented the number of audit messages lost since the last time the message came out or the last time an audit record came out. Today its a cumulative count since the system was booted. Is it too much overhead to zero it?Shouldn't be too much overhead, we are already on a slow/unlikely path. What's the benefit though? Just don't want to have to do a subtraction?
Well that, plus if the system is up for a long time (which we hope) and the message is infrequent (which we also hope), then it could take me a while to find the previous message in order to do the subtraction.
Quoted 8 lines
Quoted 3 lines
If we are dropping the 'we lost some messages' message 0'ing the counter at that time would be a bad idea, certainly not unsolvable, but I don't see what it buys us.
I wouldn't want to lose the message, just make it more useful. And if we zero it we don't have to worry about it wrapping. As it is now, its really just the count since the last time it wrapped.
Quoted 8 lines
I like Linda's idea of zero'ing the lost message counter once we are able to start sending messages again for all the reasons listed above. I haven't looked at the audit message sending code, but we are only talking about adding an extra conditional in the common case and in the worst case a conditional and an assignment. Granted they are atomic ops, but everyone keeps telling me that atomic ops are pretty quick on almost all of the platforms that Linux supports ...
Quoted 5 lines
Delivery of audit lost messages is through printk/syslog. Assuming we can assure it gets out of printk when we reset the counter we can't assure that it made it to syslog. That means we could lose that message and have no record of it at all, nor any chance that in the future it would get recorded that it was lost either.
Quoted 4 lines
That sort of begs the question - why do we even bother printing the audit record lost message?
:)
Hey its best effort what can I say. At least without reseting the counter we could realize one of them didn't make it sometime later. Not
worth much I admit :)
-Eric