audit: add backlog high water mark metric
PATCH 1/1 audit: add backlog high water mark metric
Currently, determining the optimal audit_backlog_limit relies on instantaneous polling of the queue size. This misses transient micro-bursts, making it difficult for system administrators to know if their queue is adequately sized or if they are at risk of dropping events.
This patch introduces backlog_max_depth, a high-water mark metric that tracks the maximum number of buffers in the audit queue since the system was booted or the metric was last reset. To minimize performance overhead in the fast-path, the metric is updated using a lockless cmpxchg loop in __audit_log_end().
Userspace can read-and-clear this metric by sending an AUDIT_SET message with the AUDIT_STATUS_BACKLOG_MAX_DEPTH mask. To support periodic telemetry polling (e.g., statsd, Prometheus), the reset operation atomically returns the snapshot of the high-water mark right before zeroing it, ensuring no peaks are lost between polls.
--- include/linux/audit.h | 3 ++- include/uapi/linux/audit.h | 2 ++ kernel/audit.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-)
include/linux/audit.h +2 -1
@@ -22,7 +22,8 @@ AUDIT_STATUS_BACKLOG_LIMIT | \ AUDIT_STATUS_BACKLOG_WAIT_TIME | \ AUDIT_STATUS_LOST | \ - AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL) + AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL | \ + AUDIT_STATUS_BACKLOG_MAX_DEPTH)#define AUDIT_INO_UNSET ((unsigned long)-1)#define AUDIT_DEV_UNSET ((dev_t)-1)
include/uapi/linux/audit.h +2 -0
@@ -355,6 +355,7 @@ enum {#define AUDIT_STATUS_BACKLOG_WAIT_TIME 0x0020#define AUDIT_STATUS_LOST 0x0040#define AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL 0x0080 +#define AUDIT_STATUS_BACKLOG_MAX_DEPTH 0x0100#define AUDIT_FEATURE_BITMAP_BACKLOG_LIMIT 0x00000001#define AUDIT_FEATURE_BITMAP_BACKLOG_WAIT_TIME 0x00000002
@@ -486,6 +487,7 @@ struct audit_status {__u32 backlog_wait_time_actual;/* time spent waiting while * message limit exceeded */ +__u32 backlog_max_depth;/* message queue max depth */ };struct audit_features {
kernel/audit.c +32 -0
@@ -163,6 +163,9 @@ static struct sk_buff_head audit_retry_queue;/* queue msgs waiting for new auditd connection */ static struct sk_buff_head audit_hold_queue; +/* audit queue high water mark since last startup or reset */ +static atomic_t audit_backlog_max_depth__read_mostly =ATOMIC_INIT (0 ); +/* queue servicing thread */ static struct task_struct *kauditd_task;static DECLARE_WAIT_QUEUE_HEAD (kauditd_wait);
@@ -1286,6 +1289,7 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh, s.backlog =skb_queue_len (&audit_queue); s.feature_bitmap =AUDIT_FEATURE_BITMAP_ALL ; s.backlog_wait_time = audit_backlog_wait_time; + s.backlog_max_depth =atomic_read (&audit_backlog_max_depth); s.backlog_wait_time_actual =atomic_read (&audit_backlog_wait_time_actual);audit_send_reply (skb, seq,AUDIT_GET ,0 ,0 , &s,sizeof (s));break ;
@@ -1399,6 +1403,12 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh,audit_log_config_change ("backlog_wait_time_actual" ,0 , actual,1 );return actual; } +if (s.mask ==AUDIT_STATUS_BACKLOG_MAX_DEPTH ) { +u32 old_depth =atomic_xchg (&audit_backlog_max_depth,0 ); + +audit_log_config_change ("backlog_max_depth" ,0 , old_depth,1 ); +return old_depth; + }break ; }case AUDIT_GET_FEATURE :
@@ -2761,6 +2771,25 @@ int audit_signal_info(int sig, struct task_struct *t)return audit_signal_info_syscall (t); } +/* +* audit_update_backlog_max_depth - update the audit queue high water mark +* +* Safely updates the audit_backlog_max_depth metric using a lockless +* cmpxchg loop. This ensures the high-water mark is accurately tracked +* even when multiple CPUs are logging audit records concurrently. +*/ +static inline void audit_update_backlog_max_depth (void ) +{ +u32 q_len =skb_queue_len (&audit_queue); +u32 q_max =atomic_read (&audit_backlog_max_depth); + +while (unlikely (q_len > q_max)) { +if (likely (atomic_try_cmpxchg (&audit_backlog_max_depth, + &q_max, q_len))) +break ; + } +} +/** * __audit_log_end - enqueue one audit record * @skb: the buffer to send
@@ -2777,6 +2806,9 @@ static void __audit_log_end(struct sk_buff *skb)/* queue the netlink packet */ skb_queue_tail (&audit_queue, skb); + +/* update backlog high water mark */ +audit_update_backlog_max_depth (); }else {audit_log_lost ("rate limit exceeded" );kfree_skb (skb);
Signature · 1 line
2.53.0
Quoted 12 lines · Eastern Daylight Time Ricardo Robaina
Currently, determining the optimal
audit_backlog_limitrelies on instantaneous polling of the queue size. This misses transient micro-bursts, making it difficult for system administrators to know if their queue is adequately sized or if they are at risk of dropping events.This patch introduces
backlog_max_depth, a high-water mark metric that tracks the maximum number of buffers in the audit queue since the system was booted or the metric was last reset. To minimize performance overhead in the fast-path, the metric is updated using a lockless cmpxchg loop in__audit_log_end().
Quoted 5 lines
Userspace can read-and-clear this metric by sending an
AUDIT_SETmessage with theAUDIT_STATUS_BACKLOG_MAX_DEPTHmask. To support periodic telemetry polling (e.g., statsd, Prometheus), the reset operation atomically returns the snapshot of the high-water mark right before zeroing it, ensuring no peaks are lost between polls.
From a user space point of view, this looks good. User space support was co- developed alongside of this patch to ensure it works as advertised.
-Steve
Quoted 67 lines
Suggested-by Steve Grubb <[email protected]>Signed-off-by Ricardo Robaina <[email protected]>--- include/linux/audit.h | 3 ++- include/uapi/linux/audit.h | 2 ++ kernel/audit.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-)include/linux/audit.h +2
-1@@ -22,7 +22,8 @@ AUDIT_STATUS_BACKLOG_LIMIT | \ AUDIT_STATUS_BACKLOG_WAIT_TIME | \ AUDIT_STATUS_LOST | \ - AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL) + AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL | \ + AUDIT_STATUS_BACKLOG_MAX_DEPTH)#define AUDIT_INO_UNSET ((unsigned long)-1)#define AUDIT_DEV_UNSET ((dev_t)-1)include/uapi/linux/audit.h +2
-0@@ -355,6 +355,7 @@ enum {#define AUDIT_STATUS_BACKLOG_WAIT_TIME 0x0020#define AUDIT_STATUS_LOST 0x0040#define AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL 0x0080 +#define AUDIT_STATUS_BACKLOG_MAX_DEPTH 0x0100#define AUDIT_FEATURE_BITMAP_BACKLOG_LIMIT 0x00000001#define AUDIT_FEATURE_BITMAP_BACKLOG_WAIT_TIME 0x00000002@@ -486,6 +487,7 @@ struct audit_status {__u32 backlog_wait_time_actual;/* time spent waiting while * message limit exceeded */ +__u32 backlog_max_depth;/* message queue max depth */ };struct audit_features {kernel/audit.c +4
-0@@ -163,6 +163,9 @@ static struct sk_buff_head audit_retry_queue;/* queue msgs waiting for new auditd connection */ static struct sk_buff_head audit_hold_queue; +/* audit queue high water mark since last startup or reset */ +static atomic_t audit_backlog_max_depth__read_mostly =ATOMIC_INIT (0 ); +/* queue servicing thread */ static struct task_struct *kauditd_task;static DECLARE_WAIT_QUEUE_HEAD (kauditd_wait);@@ -1286,6 +1289,7 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh, s.backlog = skb_queue_len(&audit_queue); s.feature_bitmap =AUDIT_FEATURE_BITMAP_ALL ; s.backlog_wait_time = audit_backlog_wait_time; + s.backlog_max_depth =atomic_read (&audit_backlog_max_depth); s.backlog_wait_time_actual =atomic_read (&audit_backlog_wait_time_actual);audit_send_reply (skb, seq,AUDIT_GET ,0 ,0 , &s,sizeof (s));kernel/audit.c +0
-0break ;
kernel/audit.c static int audit_receive_msg(struct sk_buff *skb, · line 1409
Quoted 7 lines
kernel/audit.c +4
-0@@ -1399,6 +1403,12 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh, audit_log_config_change("backlog_wait_time_actual", 0, actual, 1); return actual; } +if (s.mask ==AUDIT_STATUS_BACKLOG_MAX_DEPTH ) { +u32 old_depth =atomic_xchg (&audit_backlog_max_depth,0 ); + + audit_log_config_change("backlog_max_depth", 0, old_depth,
1);
Quoted 41 lines
kernel/audit.c +24
-0+return old_depth; + }break ; }case AUDIT_GET_FEATURE :@@ -2761,6 +2771,25 @@ int audit_signal_info(int sig, struct task_struct *t) return audit_signal_info_syscall(t); } +/* +* audit_update_backlog_max_depth - update the audit queue high water mark +* +* Safely updates the audit_backlog_max_depth metric using a lockless +* cmpxchg loop. This ensures the high-water mark is accurately tracked +* even when multiple CPUs are logging audit records concurrently. +*/ +static inline void audit_update_backlog_max_depth (void ) +{ +u32 q_len =skb_queue_len (&audit_queue); +u32 q_max =atomic_read (&audit_backlog_max_depth); + +while (unlikely (q_len > q_max)) { +if (likely (atomic_try_cmpxchg (&audit_backlog_max_depth, + &q_max, q_len))) +break ; + } +} +/** * __audit_log_end - enqueue one audit record * @skb: the buffer to send @@ -2777,6 +2806,9 @@ static void __audit_log_end(struct sk_buff *skb)/* queue the netlink packet */ skb_queue_tail (&audit_queue, skb); + +/* update backlog high water mark */ +audit_update_backlog_max_depth (); }else {audit_log_lost ("rate limit exceeded" );kfree_skb (skb);
Paul Moore
maintainerQuoted 19 lines · Ricardo Robaina
Currently, determining the optimal
audit_backlog_limitrelies on instantaneous polling of the queue size. This misses transient micro-bursts, making it difficult for system administrators to know if their queue is adequately sized or if they are at risk of dropping events.This patch introduces
backlog_max_depth, a high-water mark metric that tracks the maximum number of buffers in the audit queue since the system was booted or the metric was last reset. To minimize performance overhead in the fast-path, the metric is updated using a lockless cmpxchg loop in__audit_log_end().Userspace can read-and-clear this metric by sending an
AUDIT_SETmessage with theAUDIT_STATUS_BACKLOG_MAX_DEPTHmask. To support periodic telemetry polling (e.g., statsd, Prometheus), the reset operation atomically returns the snapshot of the high-water mark right before zeroing it, ensuring no peaks are lost between polls.
Quoted 8 lines
Suggested-by Steve Grubb <[email protected]>Signed-off-by Ricardo Robaina <[email protected]>--- include/linux/audit.h | 3 ++- include/uapi/linux/audit.h | 2 ++ kernel/audit.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-)
I sat on this for a bit because I wanted to think on it for a while. While I agree audit could benefit from better statistics around queue/backlog status, I'm not sure a single "max" value alone is worth a bit in the audit_status bitmask. My concern is that the max queue length only provides a single snapshot of what the queue looked like, it doesn't give any indication of the average queue length over a span of time. Some audit users are willing to live with occasional drops, and the max size doesn't help them arrive at a good balance.
As for the users who can't tolerate any audit record drops? They shouldn't be running with a backlog limit anyway so the maximum queue value will be of limit use.
Signature · 1 line
paul-moore.com
Hello Paul,
Quoted 1 line · Eastern Daylight Time Paul Moore
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina <[email protected]>
Quoted 37 lines · wrote:
Quoted 21 lines
Currently, determining the optimal
audit_backlog_limitrelies on instantaneous polling of the queue size. This misses transient micro-bursts, making it difficult for system administrators to know if their queue is adequately sized or if they are at risk of dropping events.This patch introduces
backlog_max_depth, a high-water mark metric that tracks the maximum number of buffers in the audit queue since the system was booted or the metric was last reset. To minimize performance overhead in the fast-path, the metric is updated using a lockless cmpxchg loop in__audit_log_end().Userspace can read-and-clear this metric by sending an
AUDIT_SETmessage with theAUDIT_STATUS_BACKLOG_MAX_DEPTHmask. To support periodic telemetry polling (e.g., statsd, Prometheus), the reset operation atomically returns the snapshot of the high-water mark right before zeroing it, ensuring no peaks are lost between polls.Suggested-by Steve Grubb <[email protected]>Signed-off-by Ricardo Robaina <[email protected]>Quoted 6 lines
---
include/linux/audit.h | 3 ++- include/uapi/linux/audit.h | 2 ++ kernel/audit.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-)I sat on this for a bit because I wanted to think on it for a while. While I agree audit could benefit from better statistics around queue/backlog status, I'm not sure a single "max" value alone is worth a bit in the
audit_statusbitmask. My concern is that the max queue length only provides a single snapshot of what the queue looked like, it doesn't give any indication of the average queue length over a span of time. Some audit users are willing to live with occasional drops, and the max size doesn't help them arrive at a good balance.
Quoted 3 lines
As for the users who can't tolerate any audit record drops? They shouldn't be running with a backlog limit anyway so the maximum queue value will be of limit use.
The existing audit_lost counter tells administrators they have already failed; the proposed backlog_max_depth tells them they are at risk of failing. These are different signals serving different operational needs. The dominant real-world deployment — compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events — has no existing mechanism to verify their limit is correctly sized between polling intervals. Instantaneous backlog polling is blind to sub-second bursts. Only a high-water mark, atomically reset at each poll, closes this gap. The average queue length would not answer the question 'did I ever come close to the limit?' — only the maximum can.
On the bitmask concern: the last addition was AUDIT_STATUS_BACKLOG_WAIT_TIME_ACTUAL, six years ago.
If you don't think this closes the gap on what people need, the patch could be amended to include backlog_lost_since_reset (drops since last poll) alongside the max so that you get two metrics for the price of one bit. But this is absolutely needed because people are flying blind without it.
-Steve
Paul Moore
maintainerQuoted 4 lines · Steve Grubb
Quoted 1 line · Eastern Daylight Time Paul Moore
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina <[email protected]>
wrote:
...
Quoted 1 line
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
Signature · 1 line
paul-moore.com
Paul Moore
maintainerQuoted 6 lines · Paul Moore
Quoted 4 lines · Steve Grubb
Quoted 1 line · Eastern Daylight Time Paul Moore
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina <[email protected]>
wrote:
Quoted 7 lines
...
Quoted 1 line
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
To be perfectly honest, it's also impossible to require zero dropped events. Even in the most extreme configurations where the admin decides to panic the system, that only happens once the system reaches the point where it is dropping events. We try *really* hard to not drop events, but it is always going to be a possibility.
Signature · 1 line
paul-moore.com
Quoted 3 lines · Eastern Daylight Time Paul Moore
Quoted 2 lines · Paul Moore
Quoted 1 line · Steve Grubb
On Friday, April 10, 2026 5:34:08 PM Eastern Daylight Time Paul Moore
Quoted 12 lines · wrote:
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina
<[email protected]>wrote:
...
Quoted 2 lines
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...>
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
Quoted 5 lines
To be perfectly honest, it's also impossible to require zero dropped events. Even in the most extreme configurations where the admin decides to panic the system, that only happens once the system reaches the point where it is dropping events. We try *really* hard to not drop events, but it is always going to be a possibility.
You're helping make the point. Those administrators have decided reliable auditing is more important than system availability. backlog_max_depth gives them the one thing they currently lack: advance warning. If the high-water mark is consistently approaching the backlog limit, they have actionable information to raise the limit, reduce audit rule coverage, or address the underlying load - before the system goes down. These are exactly the users who would benefit the most from this metric.
-Steve
Paul Moore
maintainerQuoted 18 lines · Steve Grubb
Quoted 3 lines · Eastern Daylight Time Paul Moore
Quoted 2 lines · Paul Moore
Quoted 1 line · Steve Grubb
On Friday, April 10, 2026 5:34:08 PM Eastern Daylight Time Paul Moore
Quoted 12 lines · wrote:
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina
<[email protected]>wrote:
...
Quoted 2 lines
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...>
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
Quoted 8 lines
Quoted 5 lines
To be perfectly honest, it's also impossible to require zero dropped events. Even in the most extreme configurations where the admin decides to panic the system, that only happens once the system reaches the point where it is dropping events. We try *really* hard to not drop events, but it is always going to be a possibility.
You're helping make the point. Those administrators have decided reliable auditing is more important than system availability.
Users prioritizing reliable auditing over system availability should not run with a backlog limit. It's that simple.
Regardless, I'm still not convinced this maximum backlog stat alone will solve any meaningful problems. If your audit log is predictable enough that this metric has value, it should be possible to either capture the backlog size during periods of high audit load or simply run the system through that load and verify it doesn't crash and go to hell. If your audit log isn't predictable, capturing a maximum backlog size doesn't really mean anything since it is still a snapshot of one instance of the system and there is always the possibility of the system exceeding it.
Signature · 1 line
paul-moore.com
Paul Moore
maintainerQuoted 25 lines · Paul Moore
Quoted 24 lines · Steve Grubb
Quoted 3 lines · Eastern Daylight Time Paul Moore
Quoted 2 lines · Paul Moore
Quoted 1 line · Steve Grubb
On Friday, April 10, 2026 5:34:08 PM Eastern Daylight Time Paul Moore
Quoted 17 lines · wrote:
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina
<[email protected]>wrote:
...
Quoted 2 lines
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...>
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
To be perfectly honest, it's also impossible to require zero dropped events. Even in the most extreme configurations where the admin decides to panic the system, that only happens once the system reaches the point where it is dropping events. We try *really* hard to not drop events, but it is always going to be a possibility.
Quoted 5 lines
Quoted 2 lines
You're helping make the point. Those administrators have decided reliable auditing is more important than system availability.
Users prioritizing reliable auditing over system availability should not run with a backlog limit. It's that simple.
To clarify this further, even on systems without a backlog limit and a panic-on-loss configuration, there is still a possibility that the system could lose an event when it hits the edge before it panics. A maximum backlog stat won't help here. Even if you had a way to capture the backlog size before the system took itself out, the size is flirting with the maximum resource limits of the system, it would be silly to use that as a configured backlog limit, you would still want to leave the limit at 0/disabled.
Quoted 9 lines
Regardless, I'm still not convinced this maximum backlog stat alone will solve any meaningful problems. If your audit log is predictable enough that this metric has value, it should be possible to either capture the backlog size during periods of high audit load or simply run the system through that load and verify it doesn't crash and go to hell. If your audit log isn't predictable, capturing a maximum backlog size doesn't really mean anything since it is still a snapshot of one instance of the system and there is always the possibility of the system exceeding it.
Signature · 1 line
paul-moore.com
Quoted 52 lines · Paul Moore
Quoted 25 lines · Paul Moore
Quoted 24 lines · Steve Grubb
Quoted 3 lines · Eastern Daylight Time Paul Moore
Quoted 2 lines · Paul Moore
Quoted 1 line · Steve Grubb
On Friday, April 10, 2026 5:34:08 PM Eastern Daylight Time Paul Moore
Quoted 17 lines · wrote:
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina
<[email protected]>wrote:
...
Quoted 2 lines
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...>
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
To be perfectly honest, it's also impossible to require zero dropped events. Even in the most extreme configurations where the admin decides to panic the system, that only happens once the system reaches the point where it is dropping events. We try *really* hard to not drop events, but it is always going to be a possibility.
Quoted 5 lines
Quoted 2 lines
You're helping make the point. Those administrators have decided reliable auditing is more important than system availability.
Users prioritizing reliable auditing over system availability should not run with a backlog limit. It's that simple.
To clarify this further, even on systems without a backlog limit and a panic-on-loss configuration, there is still a possibility that the system could lose an event when it hits the edge before it panics. A maximum backlog stat won't help here. Even if you had a way to capture the backlog size before the system took itself out, the size is flirting with the maximum resource limits of the system, it would be silly to use that as a configured backlog limit, you would still want to leave the limit at 0/disabled.
Quoted 9 lines
Regardless, I'm still not convinced this maximum backlog stat alone will solve any meaningful problems. If your audit log is predictable enough that this metric has value, it should be possible to either capture the backlog size during periods of high audit load or simply run the system through that load and verify it doesn't crash and go to hell. If your audit log isn't predictable, capturing a maximum backlog size doesn't really mean anything since it is still a snapshot of one instance of the system and there is always the possibility of the system exceeding it.
Quoted 3 lines
--
paul-moore.com
Hi Paul,
Thanks for reviewing the patch and giving your perspective on it.
I understand your point that if a system truly prioritizes auditing over everything else, it shouldn't run with a limit. However, in practice, there is a middle ground where compliance frameworks or internal infrastructure policies require a finite backlog limit to ensure memory safety, while still demanding reliable auditing.
Currently, audit users are looking for a way to tune the system based on an optimal setting for their workload that satisfies memory constraints while practically minimizing dropped events to near-zero. I strongly believe such users would make good use of backlog_max_depth because it lets them know what the worst-case scenarios look like and how big the spikes can be. This allows them to base their tuning decisions on real data rather than guesswork, as is usually done nowadays. Other than that, exposing such metrics would allow users to leverage services like tuned to dynamically adjust limits based on workload.
That being said, I hear your concern about whether a single "max" value alone is worth consuming a bit in the audit_status bitmask. So, I'd like to ask what specific metric or combination of metrics you would be willing to consider? You mentioned average queue length earlier, and Steve suggested combining the max depth with a backlog_lost_since_reset counter. I'm happy to work on a v2 that addresses your concerns while still delivering the metrics audit users currently lack.
-Ricardo
Paul Moore
maintainerQuoted 58 lines · Ricardo Robaina
Quoted 51 lines · Paul Moore
Quoted 25 lines · Paul Moore
Quoted 24 lines · Steve Grubb
Quoted 3 lines · Eastern Daylight Time Paul Moore
Quoted 2 lines · Paul Moore
Quoted 1 line · Steve Grubb
On Friday, April 10, 2026 5:34:08 PM Eastern Daylight Time Paul Moore
Quoted 17 lines · wrote:
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
On Mon, Mar 23, 2026 at 11:07 AM Ricardo Robaina
<[email protected]>wrote:
...
Quoted 2 lines
... compliance-driven systems that must use a finite backlog limit for memory safety but cannot tolerate dropped events ...>
You must pick one of those two requirements, or at the very least prioritize them; it is simply impossible to both limit the backlog queue and require zero dropped events.
To be perfectly honest, it's also impossible to require zero dropped events. Even in the most extreme configurations where the admin decides to panic the system, that only happens once the system reaches the point where it is dropping events. We try *really* hard to not drop events, but it is always going to be a possibility.
Quoted 5 lines
Quoted 2 lines
You're helping make the point. Those administrators have decided reliable auditing is more important than system availability.
Users prioritizing reliable auditing over system availability should not run with a backlog limit. It's that simple.
To clarify this further, even on systems without a backlog limit and a panic-on-loss configuration, there is still a possibility that the system could lose an event when it hits the edge before it panics. A maximum backlog stat won't help here. Even if you had a way to capture the backlog size before the system took itself out, the size is flirting with the maximum resource limits of the system, it would be silly to use that as a configured backlog limit, you would still want to leave the limit at 0/disabled.
Quoted 9 lines
Regardless, I'm still not convinced this maximum backlog stat alone will solve any meaningful problems. If your audit log is predictable enough that this metric has value, it should be possible to either capture the backlog size during periods of high audit load or simply run the system through that load and verify it doesn't crash and go to hell. If your audit log isn't predictable, capturing a maximum backlog size doesn't really mean anything since it is still a snapshot of one instance of the system and there is always the possibility of the system exceeding it.
Quoted 3 lines
--
paul-moore.comHi Paul,
Quoted 7 lines
Thanks for reviewing the patch and giving your perspective on it.
I understand your point that if a system truly prioritizes auditing over everything else, it shouldn't run with a limit. However, in practice, there is a middle ground where compliance frameworks or internal infrastructure policies require a finite backlog limit to ensure memory safety, while still demanding reliable auditing.
It is important that those users understand they are believing a lie if they think one can demand reliable auditing with a finite backlog limit.
Quoted 6 lines
I'd like to ask what specific metric or combination of metrics you would be willing to consider? You mentioned average queue length earlier, and Steve suggested combining the max depth with a
backlog_lost_since_resetcounter. I'm happy to work on a v2 that addresses your concerns while still delivering the metrics audit users currently lack.
My suggestion would be to put forth a proposal explaining the problems you want to solve and what metrics you believe are important towards solving those problems. I agree that the current list of audit metrics are rather sparse, but as we've seen here, I don't think we yet have agreement on what metrics would be useful. My hope is that having a discussion on the metrics first could avoid false starts as we've seen here.
Signature · 1 line
paul-moore.com