trace, livepatch: Allow kprobe return overriding for livepatched functions
Subsystems (8)
cover trace, livepatch: Allow kprobe return overriding for livepatched functions
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.
A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.
To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.
Furthermore, this mechanism provides a lower-maintenance alternative to out-of-tree BPF hooks. Given the complexities of upstreaming custom BPF hooks (e.g., [0], [1]), this hybrid mode allows for the maintenance of stable, minimal hook points via livepatching with significantly reduced maintenance burden.
Yafang Shao (4):
trace: Simplify kprobe overridable function check
trace: Allow kprobes to override livepatched functions
livepatch: Add "replaceable" attribute to klp_patch
livepatch: Implement livepatch hybrid mode
include/linux/livepatch.h | 2 ++ kernel/livepatch/core.c | 50 +++++++++++++++++++++++++++++++ kernel/trace/Kconfig | 14 +++++++++ kernel/trace/bpf_trace.c | 14 ++++++--- kernel/trace/trace_kprobe.c | 49 ++++++++++++------------------ kernel/trace/trace_probe.h | 59 +++++++++++++++++++++++++++---------- 6 files changed, 139 insertions(+), 49 deletions(-)
Signature · 1 line
2.47.3
RFC PATCH 1/4 trace: Simplify kprobe overridable function check
Simplify the logic for checking overridable kprobe functions by removing redundant code.
No functional change.
--- kernel/trace/bpf_trace.c | 13 ++++++--- kernel/trace/trace_kprobe.c | 40 +++++---------------------- kernel/trace/trace_probe.h | 54 ++++++++++++++++++++++++++----------- 3 files changed, 54 insertions(+), 53 deletions(-)
kernel/trace/bpf_trace.c +9 -4
@@ -1929,10 +1929,15 @@ int perf_event_attach_bpf_prog(struct perf_event *event,* Kprobe override only works if they are on the function entry, * and only if they are on the opt-in list. */ -if (prog->kprobe_override && - (!trace_kprobe_on_func_entry (event->tp_event) || - !trace_kprobe_error_injectable (event->tp_event))) -return -EINVAL ; +if (prog->kprobe_override) { +struct trace_kprobe *tp =trace_kprobe_primary_from_call (event->tp_event); + +if (!tp) +return -EINVAL ; +if (!trace_kprobe_on_func_entry (tp) || + !trace_kprobe_error_injectable (tp)) +return -EINVAL ; + }mutex_lock (&bpf_event_mutex);
kernel/trace/trace_kprobe.c +6 -34
@@ -53,17 +53,6 @@ static struct dyn_event_operations trace_kprobe_ops = { .match = trace_kprobe_match, }; -/* -* Kprobe event core functions -*/ -struct trace_kprobe { -struct dyn_event devent; -struct kretprobe rp;/* Use rp.kp for kprobe use */ -unsigned long __percpu *nhit; -const char *symbol;/* symbol name */ -struct trace_probe tp; -}; -static bool is_trace_kprobe (struct dyn_event *ev) {return ev->ops == &trace_kprobe_ops;
@@ -212,33 +201,16 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)return addr; } -static nokprobe_inline struct trace_kprobe * -trace_kprobe_primary_from_call (struct trace_event_call *call) -{ -struct trace_probe *tp; - - tp =trace_probe_primary_from_call (call); -if (WARN_ON_ONCE (!tp)) -return NULL ; - - return container_of(tp, struct trace_kprobe, tp); -} - -bool trace_kprobe_on_func_entry (struct trace_event_call *call) +bool trace_kprobe_on_func_entry (struct trace_kprobe *tp) { -struct trace_kprobe *tk =trace_kprobe_primary_from_call (call); - -return tk ? (kprobe_on_func_entry (tk->rp.kp.addr, - tk->rp.kp.addr ?NULL : tk->rp.kp.symbol_name, - tk->rp.kp.addr ?0 : tk->rp.kp.offset) ==0 ) : false; +return !kprobe_on_func_entry (tp->rp.kp.addr, + tp->rp.kp.addr ?NULL : tp->rp.kp.symbol_name, + tp->rp.kp.addr ?0 : tp->rp.kp.offset); } -bool trace_kprobe_error_injectable (struct trace_event_call *call) +bool trace_kprobe_error_injectable (struct trace_kprobe *tp) { -struct trace_kprobe *tk =trace_kprobe_primary_from_call (call); - -return tk ?within_error_injection_list (trace_kprobe_address (tk)) : - false; +return within_error_injection_list (trace_kprobe_address (tp)); }static int register_kprobe_event (struct trace_kprobe *tk);
kernel/trace/trace_probe.h +39 -15
78 lines not on this page · raw message
Signature · 1 line
2.47.3
Re: [RFC PATCH 1/4] trace: Simplify kprobe overridable function check
Quoted 4 lines · On Thu, 2 Apr 2026 17:26:04 +0800 Yafang Shao <[email protected]> wrote:
Simplify the logic for checking overridable kprobe functions by removing redundant code.
No functional change.
NACK.
trace_kprobe must be hidden inside the trace_kprobe.c. It is not designed to be exposed.
Thank you,
Quoted 183 lines
Signed-off-by Yafang Shao <[email protected]>--- kernel/trace/bpf_trace.c | 13 ++++++--- kernel/trace/trace_kprobe.c | 40 +++++---------------------- kernel/trace/trace_probe.h | 54 ++++++++++++++++++++++++++----------- 3 files changed, 54 insertions(+), 53 deletions(-)kernel/trace/bpf_trace.c +9
-4@@ -1929,10 +1929,15 @@ int perf_event_attach_bpf_prog(struct perf_event *event,* Kprobe override only works if they are on the function entry, * and only if they are on the opt-in list. */ -if (prog->kprobe_override && - (!trace_kprobe_on_func_entry (event->tp_event) || - !trace_kprobe_error_injectable (event->tp_event))) -return -EINVAL ; +if (prog->kprobe_override) { +struct trace_kprobe *tp =trace_kprobe_primary_from_call (event->tp_event); + +if (!tp) +return -EINVAL ; +if (!trace_kprobe_on_func_entry (tp) || + !trace_kprobe_error_injectable (tp)) +return -EINVAL ; + }mutex_lock (&bpf_event_mutex);kernel/trace/trace_kprobe.c +6
-34@@ -53,17 +53,6 @@ static struct dyn_event_operations trace_kprobe_ops = { .match = trace_kprobe_match, }; -/* -* Kprobe event core functions -*/ -struct trace_kprobe { -struct dyn_event devent; -struct kretprobe rp;/* Use rp.kp for kprobe use */ -unsigned long __percpu *nhit; -const char *symbol;/* symbol name */ -struct trace_probe tp; -}; -static bool is_trace_kprobe (struct dyn_event *ev) {return ev->ops == &trace_kprobe_ops;@@ -212,33 +201,16 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)return addr; } -static nokprobe_inline struct trace_kprobe * -trace_kprobe_primary_from_call (struct trace_event_call *call) -{ -struct trace_probe *tp; - - tp =trace_probe_primary_from_call (call); -if (WARN_ON_ONCE (!tp)) -return NULL ; - - return container_of(tp, struct trace_kprobe, tp); -} - -bool trace_kprobe_on_func_entry (struct trace_event_call *call) +bool trace_kprobe_on_func_entry (struct trace_kprobe *tp) { -struct trace_kprobe *tk =trace_kprobe_primary_from_call (call); - -return tk ? (kprobe_on_func_entry (tk->rp.kp.addr, - tk->rp.kp.addr ?NULL : tk->rp.kp.symbol_name, - tk->rp.kp.addr ?0 : tk->rp.kp.offset) ==0 ) : false; +return !kprobe_on_func_entry (tp->rp.kp.addr, + tp->rp.kp.addr ?NULL : tp->rp.kp.symbol_name, + tp->rp.kp.addr ?0 : tp->rp.kp.offset); } -bool trace_kprobe_error_injectable (struct trace_event_call *call) +bool trace_kprobe_error_injectable (struct trace_kprobe *tp) { -struct trace_kprobe *tk =trace_kprobe_primary_from_call (call); - -return tk ?within_error_injection_list (trace_kprobe_address (tk)) : - false; +return within_error_injection_list (trace_kprobe_address (tp)); }static int register_kprobe_event (struct trace_kprobe *tk);kernel/trace/trace_probe.h +39
-15@@ -30,6 +30,7 @@#include "trace.h" #include "trace_output.h" +#include "trace_dynevent.h" #define MAX_TRACE_ARGS 128#define MAX_ARGSTR_LEN 63@@ -210,21 +211,6 @@ DECLARE_BASIC_PRINT_TYPE_FUNC(symbol);#define ASSIGN_FETCH_TYPE_END {}#define MAX_ARRAY_LEN 64 -#ifdef CONFIG_KPROBE_EVENTS -bool trace_kprobe_on_func_entry (struct trace_event_call *call); -bool trace_kprobe_error_injectable (struct trace_event_call *call); -#else -static inline bool trace_kprobe_on_func_entry (struct trace_event_call *call) -{ -return false; -} - -static inline bool trace_kprobe_error_injectable (struct trace_event_call *call) -{ -return false; -} -#endif /* CONFIG_KPROBE_EVENTS */ -struct probe_arg {struct fetch_insn *code;bool dynamic;/* Dynamic array (string) is used */ @@ -271,6 +257,32 @@ struct event_file_link {struct list_head list; }; +/* +* Kprobe event core functions +*/ +struct trace_kprobe { +struct dyn_event devent; +struct kretprobe rp;/* Use rp.kp for kprobe use */ +unsigned long __percpu *nhit; +const char *symbol;/* symbol name */ +struct trace_probe tp; +}; + +#ifdef CONFIG_KPROBE_EVENTS +bool trace_kprobe_on_func_entry (struct trace_kprobe *tp); +bool trace_kprobe_error_injectable (struct trace_kprobe *tp); +#else +static inline bool trace_kprobe_on_func_entry (struct trace_kprobe *tp) +{ +return false; +} + +static inline bool trace_kprobe_error_injectable (struct trace_kprobe *tp) +{ +return false; +} +#endif /* CONFIG_KPROBE_EVENTS */ +static inline unsigned int trace_probe_load_flag (struct trace_probe *tp) {return smp_load_acquire (&tp->event->flags);@@ -329,6 +341,18 @@ trace_probe_primary_from_call(struct trace_event_call *call) return list_first_entry_or_null(&tpe->probes, struct trace_probe, list); } +static nokprobe_inline struct trace_kprobe * +trace_kprobe_primary_from_call (struct trace_event_call *call) +{ +struct trace_probe *tp; + + tp =trace_probe_primary_from_call (call); +if (WARN_ON_ONCE (!tp)) +return NULL ; + + return container_of(tp, struct trace_kprobe, tp); +} +static inline struct list_head *trace_probe_probe_list (struct trace_probe *tp) {return &tp->event->probes;Signature · 1 line
2.47.3
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
RFC PATCH 2/4 trace: Allow kprobes to override livepatched functions
In reply to Yafang ShaoIntroduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.
--- kernel/trace/Kconfig | 14 ++++++++++++++ kernel/trace/bpf_trace.c | 3 ++- kernel/trace/trace_kprobe.c | 17 +++++++++++++++++ kernel/trace/trace_probe.h | 5 +++++ 4 files changed, 38 insertions(+), 1 deletion(-)
kernel/trace/Kconfig +14 -0
@@ -1279,6 +1279,20 @@ config HIST_TRIGGERS_DEBUG If unsure, say N. +config KPROBE_OVERRIDE_KLP_FUNC + bool "Allow kprobes to override livepatched functions" + depends on KPROBES && LIVEPATCH + help + This option allows BPF programs to use kprobes to override functions + that have already been patched by Livepatch (KLP). + + Enabling this provides a mechanism to dynamically control execution + flow without requiring a reboot or a new livepatch module. It + effectively combines the persistence of livepatching with the + programmability of BPF. + + If unsure, say N. + source "kernel/trace/rv/Kconfig" endif # FTRACE
kernel/trace/bpf_trace.c +2 -1
@@ -1935,7 +1935,8 @@ int perf_event_attach_bpf_prog(struct perf_event *event,if (!tp)return -EINVAL ;if (!trace_kprobe_on_func_entry (tp) || - !trace_kprobe_error_injectable (tp)) + (!trace_kprobe_error_injectable (tp) && + !trace_kprobe_klp_func_overridable (tp)))return -EINVAL ; }
kernel/trace/trace_kprobe.c +17 -0
@@ -213,6 +213,23 @@ bool trace_kprobe_error_injectable(struct trace_kprobe *tp)return within_error_injection_list (trace_kprobe_address (tp)); } +bool trace_kprobe_klp_func_overridable (struct trace_kprobe *tp) +{ +bool overridable = false; +#ifdef CONFIG_KPROBE_OVERRIDE_KLP_FUNC +struct module *mod; +unsigned long addr; + + addr =trace_kprobe_address (tp); +rcu_read_lock (); + mod =__module_address (addr); +if (mod && mod->klp) + overridable = true; +rcu_read_unlock (); +#endif +return overridable; +} +static int register_kprobe_event (struct trace_kprobe *tk);static int unregister_kprobe_event (struct trace_kprobe *tk);
kernel/trace/trace_probe.h +5 -0
@@ -271,6 +271,7 @@ struct trace_kprobe {#ifdef CONFIG_KPROBE_EVENTS bool trace_kprobe_on_func_entry (struct trace_kprobe *tp);bool trace_kprobe_error_injectable (struct trace_kprobe *tp); +bool trace_kprobe_klp_func_overridable (struct trace_kprobe *tp);#else static inline bool trace_kprobe_on_func_entry (struct trace_kprobe *tp) {
@@ -281,6 +282,10 @@ static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp) {return false; } +static inline bool trace_kprobe_klp_func_overridable (struct trace_kprobe *tp) +{ +return false; +}#endif /* CONFIG_KPROBE_EVENTS */ static inline unsigned int trace_probe_load_flag (struct trace_probe *tp)
Signature · 1 line
2.47.3
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 3 lines · Yafang Shao
Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNCconfiguration option.
Hi, Yafang. This is a interesting idea.
For now, the bpf_override_return() can only be used on the kernel functions that allow error injection to prevent the BPF program from crash the kernel. If we use it on the kernel functions that patched by the KLP, we can crash the kernel easily by return a invalid value with bpf_override_return(), right? (Of course, we can crash the kernel easily with KLP too ;)
I haven't figure out the use case yet. Can KLP be used together with the BPF program that use bpf_override_return()? The KLP will modify the RIP on the stack, and the bpf_override_return() will modify it too. AFAIK, there can't be two ftrace_ops that both have the FTRACE_OPS_FL_IPMODIFY flag. Did I miss something?
It will be helpful for me to understand the use case if a selftests is offered :)
BTW, if we allow the usage of bpf_override_return() on the KLP patched function, we should allow the usage of BPF_MODIFY_RETURN on this case too, right?
Thanks!
Menglong Dong
Quoted 100 lines
Signed-off-by Yafang Shao <[email protected]>--- kernel/trace/Kconfig | 14 ++++++++++++++ kernel/trace/bpf_trace.c | 3 ++- kernel/trace/trace_kprobe.c | 17 +++++++++++++++++ kernel/trace/trace_probe.h | 5 +++++ 4 files changed, 38 insertions(+), 1 deletion(-)kernel/trace/Kconfig +14
-0@@ -1279,6 +1279,20 @@ config HIST_TRIGGERS_DEBUG If unsure, say N. +config KPROBE_OVERRIDE_KLP_FUNC + bool "Allow kprobes to override livepatched functions" + depends on KPROBES && LIVEPATCH + help + This option allows BPF programs to use kprobes to override functions + that have already been patched by Livepatch (KLP). + + Enabling this provides a mechanism to dynamically control execution + flow without requiring a reboot or a new livepatch module. It + effectively combines the persistence of livepatching with the + programmability of BPF. + + If unsure, say N. + source "kernel/trace/rv/Kconfig" endif # FTRACEkernel/trace/bpf_trace.c +2
-1@@ -1935,7 +1935,8 @@ int perf_event_attach_bpf_prog(struct perf_event *event,if (!tp)return -EINVAL ;if (!trace_kprobe_on_func_entry (tp) || - !trace_kprobe_error_injectable (tp)) + (!trace_kprobe_error_injectable (tp) && + !trace_kprobe_klp_func_overridable (tp)))return -EINVAL ; }kernel/trace/trace_kprobe.c +17
-0@@ -213,6 +213,23 @@ bool trace_kprobe_error_injectable(struct trace_kprobe *tp)return within_error_injection_list (trace_kprobe_address (tp)); } +bool trace_kprobe_klp_func_overridable (struct trace_kprobe *tp) +{ +bool overridable = false; +#ifdef CONFIG_KPROBE_OVERRIDE_KLP_FUNC +struct module *mod; +unsigned long addr; + + addr =trace_kprobe_address (tp); +rcu_read_lock (); + mod =__module_address (addr); +if (mod && mod->klp) + overridable = true; +rcu_read_unlock (); +#endif +return overridable; +} +static int register_kprobe_event (struct trace_kprobe *tk);static int unregister_kprobe_event (struct trace_kprobe *tk);kernel/trace/trace_probe.h +5
-0@@ -271,6 +271,7 @@ struct trace_kprobe {#ifdef CONFIG_KPROBE_EVENTS bool trace_kprobe_on_func_entry (struct trace_kprobe *tp);bool trace_kprobe_error_injectable (struct trace_kprobe *tp); +bool trace_kprobe_klp_func_overridable (struct trace_kprobe *tp);#else static inline bool trace_kprobe_on_func_entry (struct trace_kprobe *tp) {@@ -281,6 +282,10 @@ static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp) {return false; } +static inline bool trace_kprobe_klp_func_overridable (struct trace_kprobe *tp) +{ +return false; +}#endif /* CONFIG_KPROBE_EVENTS */ static inline unsigned int trace_probe_load_flag (struct trace_probe *tp)
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 6 lines · Menglong Dong
Quoted 3 lines · Yafang Shao
Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNCconfiguration option.
Quoted 8 lines
Hi, Yafang. This is a interesting idea.
For now, the
bpf_override_return()can only be used on the kernel functions that allow error injection to prevent the BPF program from crash the kernel. If we use it on the kernel functions that patched by the KLP, we can crash the kernel easily by return a invalid value withbpf_override_return(), right? (Of course, we can crash the kernel easily with KLP too ;)
Right.
Livepatch already grants the power to modify the kernel at will; allowing BPF to override a patched function simply adds a layer of runtime programmability to an existing modification.
Quoted 3 lines
I haven't figure out the use case yet. Can KLP be used together with the BPF program that use
bpf_override_return()?
The two mechanisms do not target the same entry point: whileKLP modifies the original kernel function, bpf_override_return() is applied to the newly patched function provided by the KLP module.
Quoted 4 lines
The KLP will modify
the RIP on the stack, and thebpf_override_return()will modify it too. AFAIK, there can't be twoftrace_opsthat both have theFTRACE_OPS_FL_IPMODIFYflag. Did I miss something?
Correct, but as noted, they target different functions
Quoted 3 lines
It will be helpful for me to understand the use case if a selftests is offered :)
Here is a recent use case from our production environment.
- The livepatch
drivers/net/bonding/bond_main.c +0 -0
@@ -5175,12 +5175,22 @@ int bond_update_slave_arr(struct bonding
*bond, struct slave *skipslave)
return ret;
}
+/* noclone to avoid bond_get_slave_hook.constprop.0 */ +__attribute__((__noclone__, __noinline__))
+int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count) +{
+ return -1;
+}
static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
struct sk_buff *skb, struct bond_up_slave *slaves)
{
struct slave *slave;
unsigned int count;
+ int slave_idx;
u32 hash;
hash = bond_xmit_hash(bond, skb);
@@ -5188,6 +5198,13 @@ static struct slave
*bond_xmit_3ad_xor_slave_get(struct bonding *bond,
if (unlikely(!count))
return NULL;+ /* Try BPF hook first - returns slave index directly */
+ slave_idx = bond_get_slave_hook(skb, hash, count);
+ /* If BPF hook returned valid slave index, use it */
+ if (slave_idx >= 0 && slave_idx < count) {
+ slave = slaves->arr[slave_idx];
+ return slave;
+ }
slave = slaves->arr[hash % count];
return slave;}
- The BPF program
SEC("kprobe/bond_get_slave_hook")
int BPF_KPROBE(slave_selector, struct sk_buff *skb, u32 hash, u32 count)
{
unsigned short net_hdr_off;
unsigned char *head;
struct iphdr iph;
int *slave_idx;
__u32 daddr;
__u16 proto = BPF_CORE_READ(skb, protocol);
if (proto != bpf_htons(0x0800))
return 0;
head = BPF_CORE_READ(skb, head);
net_hdr_off = BPF_CORE_READ(skb, network_header);
if (bpf_probe_read_kernel(&iph, sizeof(iph), head + net_hdr_off) != 0)
return 0;
daddr = iph.daddr;
slave_idx = bpf_map_lookup_elem(&ip_slave_map, &daddr);
if (slave_idx) {
int idx = *slave_idx;
if (idx >= 0 && idx < (int)count)
bpf_override_return(ctx, idx);
}
return 0;
}Quoted 4 lines
BTW, if we allow the usage of
bpf_override_return()on the KLP patched function, we should allow the usage ofBPF_MODIFY_RETURNon this case too, right?
It's a possibility, but I haven't tested that specifically yet.
Signature · 2 lines
Regards Yafang
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
On 2026/4/2 21:20 Yafang Shao <[email protected]> write:
Quoted 9 lines · Menglong Dong
Quoted 8 lines · Yafang Shao
Quoted 3 lines
Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNCconfiguration option.Hi, Yafang. This is a interesting idea.
[...]
Review
Quoted 7 lines
(quoted diff) +6
-0+/* noclone to avoid bond_get_slave_hook.constprop.0 */ +__attribute__((__noclone__, __noinline__)) +int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count) +{ + return -1; +}
Hi, yafang.
I see what you mean now. So you want to allow BPF program override the return of all the kernel functions in a KLP module.
I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of bpf_override_write().
What's more, this is a little weird for me. If we allow to use bpf_override_return() for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?
Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in your example? Then we can override its return directly. This is a more reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.
WDYT?
BTW, this is a BPF modification, so maybe we can use "bpf: xxx" for the title of this patch. Then, the BPF maintainers can notice this patch ;)
Thanks!
Menglong Dong
Quoted 69 lines
static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
structsk_buff*skb, structbond_up_slave*slaves){ struct slave *slave; unsigned int count;(quoted diff) +8
-0+ int slave_idx; u32 hash; hash = bond_xmit_hash(bond, skb);@@ -5188,6 +5198,13 @@ static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond, if (unlikely(!count)) return NULL; + /* Try BPF hook first - returns slave index directly */ + slave_idx = bond_get_slave_hook(skb, hash, count); + /* If BPF hook returned valid slave index, use it */ + if (slave_idx >= 0 && slave_idx < count) { + slave = slaves->arr[slave_idx]; + return slave; + } slave = slaves->arr[hash % count]; return slave;}
- The BPF program
SEC("kprobe/bond_get_slave_hook")
intBPF_KPROBE(slave_selector, struct sk_buff *skb, u32 hash, u32 count){ unsigned short net_hdr_off; unsigned char *head; struct iphdr iph; int *slave_idx; __u32 daddr; __u16 proto = BPF_CORE_READ(skb, protocol); if (proto != bpf_htons(0x0800)) return 0; head = BPF_CORE_READ(skb, head); net_hdr_off = BPF_CORE_READ(skb, network_header); if (bpf_probe_read_kernel(&iph, sizeof(iph), head + net_hdr_off) != 0) return 0; daddr = iph.daddr; slave_idx = bpf_map_lookup_elem(&ip_slave_map, &daddr); if (slave_idx) { int idx = *slave_idx; if (idx >= 0 && idx < (int)count) bpf_override_return(ctx, idx); } return 0; }Quoted 4 lines
BTW, if we allow the usage of
bpf_override_return()on the KLP patched function, we should allow the usage ofBPF_MODIFY_RETURNon this case too, right?It's a possibility, but I haven't tested that specifically yet.
Signature · 2 lines
Regards Yafang
Steven Rostedt
maintainerRe: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 3 lines · On Fri, 03 Apr 2026 18:25:59 +0800 Menglong Dong <[email protected]>…
I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of
bpf_override_write().
Right, livepatching may allow for rapid experimentation but that is not its purpose. It is for fixing production systems without having to reboot. Using BPF to change the return of a function is a huge security issue.
Quoted 5 lines
What's more, this is a little weird for me. If we allow to use
bpf_override_return()for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?
Right.
Quoted 5 lines
Can we mark the "
bond_get_slave_hook" withALLOW_ERROR_INJECTION()in your example? Then we can override its return directly. This is a more reasonable for me. WithALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.
If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.
The command line should be something like:
allow_bpf_to_rootkit_functions
-- Steve
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 26 lines · Steven Rostedt
Quoted 3 lines · On Fri, 03 Apr 2026 18:25:59 +0800 Menglong Dong <[email protected]>…
I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of
bpf_override_write().Right, livepatching may allow for rapid experimentation but that is not its purpose. It is for fixing production systems without having to reboot. Using BPF to change the return of a function is a huge security issue.
Quoted 5 lines
What's more, this is a little weird for me. If we allow to use
bpf_override_return()for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?Right.
Quoted 5 lines
Can we mark the "
bond_get_slave_hook" withALLOW_ERROR_INJECTION()in your example? Then we can override its return directly. This is a more reasonable for me. WithALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.
Quoted 8 lines
If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.
The command line should be something like:
allow_bpf_to_rootkit_functions
The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In the next revision, I will rename this to CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel command-line parameter, allow_bpf_to_rootkit_functions, to control it.
--
Regards
Yafang
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 9 lines · Yafang Shao
Quoted 9 lines
If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.
The command line should be something like:
Quoted 7 lines
Quoted 1 line
allow_bpf_to_rootkit_functionsThe feature is currently gated by
CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In the next revision, I will rename this toCONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCSand introduce a corresponding kernel command-line parameter,allow_bpf_to_rootkit_functions, to control it.
No. Even with extra config this is not ok.
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 13 lines · Alexei Starovoitov
Quoted 11 lines · Yafang Shao
Quoted 10 lines
If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.
The command line should be something like:
allow_bpf_to_rootkit_functions
Quoted 7 lines
Quoted 5 lines
The feature is currently gated by
CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In the next revision, I will rename this toCONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCSand introduce a corresponding kernel command-line parameter,allow_bpf_to_rootkit_functions, to control it.No. Even with extra config this is not ok.
I will send patch #3 and #4 as a standalone patchset to upstream the hybrid livepatch first and then figure out how to move allow_bpf_to_rootkit_functions forward.
--
Regards
Yafang
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
In reply to Menglong DongQuoted 22 lines · Menglong Dong
On 2026/4/2 21:20 Yafang Shao <[email protected]> write:
Quoted 9 lines · Menglong Dong
Quoted 8 lines · Yafang Shao
Quoted 3 lines
Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNCconfiguration option.Hi, Yafang. This is a interesting idea.
[...]
Review
Quoted 7 lines
(quoted diff) +6
-0+/* noclone to avoid bond_get_slave_hook.constprop.0 */ +__attribute__((__noclone__, __noinline__)) +int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count) +{ + return -1; +}Hi, yafang.
Quoted 6 lines
I see what you mean now. So you want to allow BPF program override the return of all the kernel functions in a KLP module.
I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of
bpf_override_write().
This feature is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option, which is disabled by default. Consequently, the user must explicitly enable this option to utilize the feature.
Quoted 5 lines
What's more, this is a little weird for me. If we allow to use
bpf_override_return()for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?
By leveraging KLP, we can rapidly deploy new features without interrupting production workloads. Accordingly, this feature is specifically targeted at KLP-patched functions to maintain that seamless delivery model.
Quoted 5 lines
Can we mark the "
bond_get_slave_hook" withALLOW_ERROR_INJECTION()in your example? Then we can override its return directly. This is a more reasonable for me. WithALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.
It is unfortunate that ALLOW_ERROR_INJECTION() is incompatible with KLP-patched functions, as this limits our ability to perform fault injection on livepatched code
Quoted 5 lines
WDYT?
BTW, this is a BPF modification, so maybe we can use "bpf: xxx" for the title of this patch. Then, the BPF maintainers can notice this patch ;)
I agree with the suggestion. I will update the subject prefix to "trace, bpf:" in the next version.
Signature · 2 lines
Regards Yafang
Miroslav Benes
maintainerRe: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
In reply to Yafang ShaoHi,
Quoted 3 lines · Yafang Shao
Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNCconfiguration option.
this is imprecise if I read the code correctly. You want to override live patch functions, not the original ones which are live patched.
I also think that if nothing else, it needs to be more specific then just checking mod->klp. It should check if a function itself in klp module is overridable to keep it as limited as possible. Even with that, the concerns expressed by the others still apply.
---
Miroslav
Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions
Quoted 11 lines · Miroslav Benes
Hi,
Quoted 3 lines · Yafang Shao
Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNCconfiguration option.this is imprecise if I read the code correctly. You want to override live patch functions, not the original ones which are live patched.
Correct. The BPF program will override the livepatched functions rather than the original ones.
Quoted 4 lines
I also think that if nothing else, it needs to be more specific then just checking
mod->klp. It should check if a function itself in klp module is overridable to keep it as limited as possible.
That is exactly what I am currently implementing in trace_kprobe_klp_func_overridable().
Quoted 2 lines
Even with that, the
concerns expressed by the others still apply.
Signature · 2 lines
Regards Yafang
RFC PATCH 3/4 livepatch: Add "replaceable" attribute to klp_patch
In reply to Yafang ShaoAdd a new replaceable attribute to allow the coexistence of both atomic-replace and non-atomic-replace livepatches. If replaceable is set to 0, the livepatch will not be replaced by a subsequent atomic-replace operation.
This is a preparatory patch for following changes.
--- include/linux/livepatch.h | 2 ++ kernel/livepatch/core.c | 44 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
include/linux/livepatch.h +2 -0
@@ -124,6 +124,7 @@ struct klp_state {* @objs: object entries for kernel objects to be patched * @states: system states that can get modified * @replace: replace all actively used patches +* @replaceable: whether this patch can be replaced or not * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries
@@ -138,6 +139,7 @@ struct klp_patch {struct klp_object *objs;struct klp_state *states;bool replace; +bool replaceable;/* internal */ struct list_head list;
kernel/livepatch/core.c +44 -0
67 lines not on this page · raw message
Signature · 1 line
2.47.3
Song Liu
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 7 lines · Yafang Shao
Add a new replaceable attribute to allow the coexistence of both atomic-replace and non-atomic-replace livepatches. If replaceable is set to 0, the livepatch will not be replaced by a subsequent atomic-replace operation.
This is a preparatory patch for following changes.
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In case we really want a feature like this, shall we add a replaceable flag to each function (klp_func)? This will give us fine granularity control. For example, user A has a non-replaceable livepatch on func_a; later user B wants to patch another version of func_a. Then some logic can detect such a conflict and reject the load of user B's livepatch. Does this make sense?
Thanks,
Song
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 8 lines · Song Liu
Quoted 6 lines · Yafang Shao
Add a new replaceable attribute to allow the coexistence of both atomic-replace and non-atomic-replace livepatches. If replaceable is set to 0, the livepatch will not be replaced by a subsequent atomic-replace operation.
Quoted 6 lines
Quoted 1 line
This is a preparatory patch for following changes.
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.
We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.
However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.
Quoted 7 lines
In case we really want a feature like this, shall we add a replaceable flag to each function (
klp_func)? This will give us fine granularity control. For example, user A has a non-replaceable livepatch onfunc_a; later user B wants to patch another version offunc_a. Then some logic can detect such a conflict and reject the load of user B's livepatch. Does this make sense?
I agree with this. The ability to reject livepatches that try to change functions that have already been patched seems like an important safeguard.
Thanks,
Dylan
Song Liu
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]
Quoted 16 lines
Quoted 4 lines
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.
We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.
Quoted 5 lines
However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.
IIUC, the plan is:
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.
With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade "off-band' livepatches, we will have to unload the old version and load the new version later.
2. Any conflict with the off-band livepatches and regular livepatches will be difficult to manage. IOW, we kind removed the benefit of cumulative livepatches. For example, what shall we do if we really need two fixes to the same kernel functions: one from the original branch, the other from the off-band branch?
Thanks,
Song
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 27 lines · Song Liu
On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]Quoted 16 lines
Quoted 4 lines
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.
We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.
Quoted 5 lines
However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.
IIUC, the plan is:
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
Quoted 5 lines
I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade "off-band' livepatches, we will have to unload the old version and load the new version later.
Right. That is how the non-atomic-replace patch works.
Quoted 2 lines
2. Any conflict with the off-band livepatches and regular livepatches will be difficult to manage.
We need to manage this conflict with a complex user script. That said, everything can be controlled from userspace.
Quoted 4 lines
IOW, we kind removed the benefit of cumulative
livepatches. For example, what shall we do if we really need two fixes to the same kernel functions: one from the original branch, the other from the off-band branch?
We run tens of livepatches on our production servers and have never run into this issue. It's an extremely rare case — and if it does happen, a user script should be able to handle it just fine.
Signature · 2 lines
Regards Yafang
Song Liu
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 34 lines · Yafang Shao
Quoted 32 lines · Song Liu
On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]Quoted 16 lines
Quoted 4 lines
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.
We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.
Quoted 5 lines
However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.
IIUC, the plan is:
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.
Quoted 5 lines
Quoted 3 lines
With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.
For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
Does this make sense?
Thanks,
Song
Joe Lawrence
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 53 lines · Song Liu
Quoted 34 lines · Yafang Shao
Quoted 32 lines · Song Liu
On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]Quoted 16 lines
Quoted 4 lines
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.
We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.
Quoted 5 lines
However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.
IIUC, the plan is:
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.Quoted 5 lines
Quoted 3 lines
With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
Quoted 6 lines
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
Does this make sense?
I haven't been following the thread carefully, but could the Livepatch system state API (see Documentation/livepatch/system-state.rst) be leveraged somehow instead of adding further replace semantics?
--
Joe
Song Liu
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]
Quoted 27 lines
Quoted 27 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
Quoted 6 lines
Quoted 2 lines
Does this make sense?
I haven't been following the thread carefully, but could the Livepatch system state API (see
Documentation/livepatch/system-state.rst) be leveraged somehow instead of adding further replace semantics?
AFAICT, system state will not help Yafang's use case.
Thanks,
Song
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 29 lines · Song Liu
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
That sounds like a viable solution. I'll look into it and see how we can implement it.
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
Does this make sense?
I haven't been following the thread carefully, but could the Livepatch system state API (see
Documentation/livepatch/system-state.rst) be leveraged somehow instead of adding further replace semantics?AFAICT, system state will not help Yafang's use case.
Right.
Signature · 2 lines
Regards Yafang
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 28 lines · Yafang Shao
Quoted 26 lines · Song Liu
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
That sounds like a viable solution. I'll look into it and see how we can implement it.
Does the following change look good to you ?
Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags
Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:
replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.
To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.
--- .../livepatch/cumulative-patches.rst | 20 +++++++----- Documentation/livepatch/livepatch.rst | 31 +++++++++++++------ include/linux/livepatch.h | 8 +++-- kernel/livepatch/core.c | 4 +++ scripts/livepatch/init.c | 6 +--- scripts/livepatch/klp-build | 11 +++++-- 6 files changed, 53 insertions(+), 27 deletions(-)
(quoted diff) +0 -0
b/Documentation/livepatch/cumulative-patches.rst
index 1931f318976a..06e90dc5967c 100644
Documentation/livepatch/cumulative-patches.rst +1 -2
@@ -12,23 +12,26 @@ modified the same function in different ways. An elegant solution comes with the feature called "Atomic Replace". It allows creation of so called "Cumulative Patches". They include all wanted changes -from all older livepatches and completely replace them in one transition. +from older livepatches with a matching tag and replace them in one transition. Usage ----- -The atomic replace can be enabled by setting "replace" flag in struct
klp_patch,
-for example::
+he atomic replace can be enabled by setting a non-zero value to the "replace" +attribute in ``struct klp_patch``. This value acts as a **replace tag**, +defining the scope of the replacement.
+
+For example::
static struct klp_patch patch = {
.mod = THIS_MODULE,
.objs = objs,
- .replace = true,
+ .replace = 1,
};
All processes are then migrated to use the code only from the new patch.
-Once the transition is finished, all older patches are automatically -disabled.
+Once the transition is finished, all older patches with the same replace tag +are automatically disabled. Patches with different tags remain active.
Ftrace handlers are transparently removed from functions that are no longer modified by the new cumulative patch.
@@ -62,9 +65,10 @@ Limitations:
------------
- Once the operation finishes, there is no straightforward way
- to reverse it and restore the replaced patches atomically. + to reverse it and restore the replaced patches (with the same tag) + atomically.
- A good practice is to set .replace flag in any released livepatch.
+ A good practice is to set a consistent .replace tag in related livepatches.
Then re-adding an older livepatch is equivalent to downgrading to that patch. This is safe as long as the livepatches do _not_ do extra modifications in (un)patching callbacks or in the module_init()
(quoted diff) +0 -0
b/Documentation/livepatch/livepatch.rst
index acb90164929e..1fc1543a22c3 100644
Documentation/livepatch/livepatch.rst +22 -9
@@ -347,15 +347,28 @@ to '0'. 5.3. Replacing -------------- -All enabled patches might get replaced by a cumulative patch that -has the .replace flag set. - -Once the new patch is enabled and the 'transition' finishes then -all the functions (struct klp_func) associated with the replaced -patches are removed from the corresponding struct klp_ops. Also -the ftrace handler is unregistered and the struct klp_ops is -freed when the related function is not modified by the new patch -and func_stack list becomes empty. +All currently enabled patches may be superseded by a cumulative patch that +has the ``.replace`` attribute enabled. The behavior of the replacement +depends on the value assigned to the replace tag: + +replace = 0 + Atomic replace is disabled. However, this patch remains eligible to be + superseded by others. + +replace > 0 + Enables tagged atomic replace. Once the new patch is enabled and the + transition finishes, the livepatching core identifies all existing + patches that share the same replace tag. + +Once the transition is complete, all functions (``struct klp_func``) +associated with the matching replaced patches are removed from the +corresponding ``struct klp_ops``. If a function is no longer modified by +the new patch and its ``func_stack`` list becomes empty, the ftrace +handler is unregistered and the ``struct klp_ops`` is freed. + +Patches with a different replace tag are not affected by this process +and remain active. This allows for the independent management and +stacking of multiple, non-conflicting livepatch sets. See Documentation/livepatch/cumulative-patches.rst for more details.
include/linux/livepatch.h +6 -2
@@ -123,7 +123,11 @@ struct klp_state {* @mod: reference to the live patch module * @objs: object entries for kernel objects to be patched * @states: system states that can get modified -* @replace: replace all actively used patches +* @replace: replace tag: +* = 0: Atomic replace is disabled; however, this patch remains +* eligible to be superseded by others. +* > 0: Atomic replace is enabled. Only existing patches with a +* matching replace tag will be superseded. * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries
@@ -137,7 +141,7 @@ struct klp_patch {struct module *mod;struct klp_object *objs;struct klp_state *states; -bool replace; +unsigned int replace;/* internal */ struct list_head list;
kernel/livepatch/core.c +0 -0
@@ -793,6 +793,8 @@ void klp_free_replaced_patches_async(struct
klp_patch *new_patch)
klp_for_each_patch_safe(old_patch, tmp_patch) {
if (old_patch == new_patch)
return;
+ if (old_patch->replace && old_patch->replace !=
new_patch->replace)
+ continue;
klp_free_patch_async(old_patch);
}
}
@@ -1194,6 +1196,8 @@ void klp_unpatch_replaced_patches(struct
klp_patch *new_patch)
klp_for_each_patch(old_patch) {
if (old_patch == new_patch)
return;+ if (old_patch->replace && old_patch->replace !=new_patch->replace)
+ continue;
old_patch->enabled = false;
klp_unpatch_objects(old_patch);scripts/livepatch/init.c +1 -5
@@ -72,11 +72,7 @@ static int __init livepatch_mod_init(void)/* TODO patch->states */ -#ifdef KLP_NO_REPLACE - patch->replace = false; -#else - patch->replace = true; -#endif + patch->replace =KLP_REPLACE_TAG ;return klp_enable_patch (patch);
scripts/livepatch/klp-build +0 -0
@@ -118,6 +118,7 @@ Options: -j, --jobs=<jobs> Build jobs to run simultaneously
[default: $JOBS]
-o, --output=<file.ko> Output file [default: livepatch-<patch-name>.ko]
--no-replace Disable livepatch atomic replace
+ -t, --replace-tag=<tag> Set the atomic replace tag for this livepatch
-v, --verbose Pass V=1 to kernel/module builds Advanced Options:
@@ -142,8 +143,8 @@ process_args() {
local long
local args
- short="hfj:o:vdS:T" - long="help,show-first-changed,jobs:,output:,no-replace,verbose,debug,short-circuit:,keep-tmp" + short="hfj:o:t:vdS:T" + long="help,show-first-changed,jobs:,output:,no-replace,replace-tag:,verbose,debug,short-circuit:,keep-tmp"
args=$(getopt --options "$short" --longoptions "$long" -- "$@") || {
echo; usage; exit
@@ -176,6 +177,10 @@ process_args() {
REPLACE=0
shift
;;
+ -t | --replace-tag)
+ REPLACE="$2"
+ shift 2
+ ;;
-v | --verbose)
VERBOSE="V=1"
shift@@ -759,7 +764,7 @@ build_patch_module() {
cflags=("-ffunction-sections")
cflags+=("-fdata-sections")- [[ $REPLACE -eq 0 ]] && cflags+=("-DKLP_NO_REPLACE")
+ cflags+=("-DKLP_REPLACE_TAG=$REPLACE")
cmd=("make")
cmd+=("$VERBOSE")--
Regards
Yafang
Petr Mladek
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 24 lines · Yafang Shao
Quoted 23 lines · Yafang Shao
Quoted 21 lines · Song Liu
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
I still think how to make the hybrid mode more secure:
+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?
Quoted 25 lines
Quoted 2 lines
That sounds like a viable solution. I'll look into it and see how we can implement it.
Does the following change look good to you ?
Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags
Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:
replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.
include/linux/livepatch.h +0
-0
include/linux/livepatch.h struct klp_state { · line 128
Quoted 7 lines
include/linux/livepatch.h +3
-1@@ -123,7 +123,11 @@ struct klp_state {* @mod: reference to the live patch module * @objs: object entries for kernel objects to be patched * @states: system states that can get modified -* @replace: replace all actively used patches +* @replace: replace tag: +* = 0: Atomic replace is disabled; however, this patch remains +* eligible to be superseded by others.
This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Do we still need this category?
include/linux/livepatch.h struct klp_patch { · line 144
Quoted 11 lines
include/linux/livepatch.h +3
-1+* > 0: Atomic replace is enabled. Only existing patches with a +* matching replace tag will be superseded. * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries @@ -137,7 +141,7 @@ struct klp_patch {struct module *mod;struct klp_object *objs;struct klp_state *states; -bool replace; +unsigned int replace;
This already breaks the backward compatibility by changing the type and semantic of this field. I would also change the name to better match the new semantic. What about using:
* @replace_set: Livepatch using the same @replace_set will get
atomically replaced, see also conflicts[*]. unsigned int replace_set;
[*] A livepatch module, livepatching an already livepatches function,
can be loaded only when it has the same @replace_set number.
By other words, two livepatches conflict when they have a different @replace_set number and have at least one livepatched version in common.
Quoted 27 lines
include/linux/livepatch.h +0
-0/* internal */ struct list_head list;kernel/livepatch/core.c +2
-0@@ -793,6 +793,8 @@ void klp_free_replaced_patches_async(struct klp_patch *new_patch) klp_for_each_patch_safe(old_patch, tmp_patch) {if (old_patch == new_patch)return ; +if (old_patch->replace && old_patch->replace != new_patch->replace) +continue ;klp_free_patch_async (old_patch);}}
kernel/livepatch/core.c +2
-0@@ -1194,6 +1196,8 @@ void klp_unpatch_replaced_patches(struct klp_patch *new_patch) klp_for_each_patch(old_patch) {if (old_patch == new_patch)return ; +if (old_patch->replace && old_patch->replace != new_patch->replace) +continue ;
Quoted 2 lines
old_patch->enabled = false;
klp_unpatch_objects(old_patch);This handles only the freeing part. More changes will be necessary:
+ klp_is_patch_compatible() must check also conflicts between livepatches with different @replace_set. The conflicts might be in the lists of:
+ livepatched functions
+ state IDs (aka callbacks and shadow variables IDs)
+ klp_add_nops() must skip livepatches with another @replace_set
+ klp_unpatch_replaced_patches() should unpatch only patches with the same @replace_set
Finally, we would need to update existing selftests plus add new selftests.
It is possible that I have missed something.
Anyway, you should wait for more feedback before you do too much coding, especially the selftests are not needed at RFC stage.
Best Regards,
Petr
Song Liu
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
On Tue, Apr 7, 2026 at 8:08 AM Petr Mladek <[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 3 lines
+ * @replace: replace tag:
+ * = 0: Atomic replace is disabled; however, this patch remains
+ * eligible to be superseded by others.This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Do we still need this category?
Quoted 14 lines
include/linux/livepatch.h struct klp_patch { · line 144
Quoted 11 lines
+ * > 0: Atomic replace is enabled. Only existing patches with a
+ * matching replace tag will be superseded.* @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entriesinclude/linux/livepatch.h +1
-1@@ -137,7 +141,7 @@ struct klp_patch {struct module *mod;struct klp_object *objs;struct klp_state *states; -bool replace; +unsigned int replace;This already breaks the backward compatibility by changing the type and semantic of this field.
I was thinking if replace=0 means no replace, it is still backward compatible. Did I miss something?
Thanks,
Song
Quoted 15 lines
I would also change the name to better
match the new semantic. What about using:* @replace_set: Livepatch using the same @replace_set will get atomically replaced, see also conflicts[*].
unsigned int
replace_set;[*] A livepatch module, livepatching an already livepatches function,
can be loaded only when it has the same @replace_set number.By other words, two livepatches conflict when they have a different @replace_set number and have at least one livepatched version in common.
[...]
Petr Mladek
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 26 lines · Song Liu
On Tue, Apr 7, 2026 at 8:08 AM Petr Mladek <[email protected]> wrote:
[...]Quoted 9 lines
Quoted 3 lines
+ * @replace: replace tag:
+ * = 0: Atomic replace is disabled; however, this patch remains
+ * eligible to be superseded by others.This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Do we still need this category?
Quoted 14 lines
include/linux/livepatch.h struct klp_patch { · line 144
Quoted 11 lines
+ * > 0: Atomic replace is enabled. Only existing patches with a
+ * matching replace tag will be superseded.* @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entriesinclude/linux/livepatch.h +1
-1@@ -137,7 +141,7 @@ struct klp_patch {struct module *mod;struct klp_object *objs;struct klp_state *states; -bool replace; +unsigned int replace;This already breaks the backward compatibility by changing the type and semantic of this field.
Quoted 2 lines
I was thinking if replace=0 means no replace, it is still backward compatible. Did I miss something?
IMHO, the semantic of the no-replace mode would be strange if we introduce the hybrid mode. Especially, it would be strange when it can be replaced by any livepatch with random replace tag/set. Also it would just complicate the definition and detection of conflicts.
I am going to provide more details in the reply to Yafang.
Best Regards,
Petr
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
In reply to Petr MladekQuoted 31 lines · Petr Mladek
Quoted 29 lines · Yafang Shao
Quoted 28 lines · Yafang Shao
Quoted 26 lines · Song Liu
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
I still think how to make the hybrid mode more secure:
+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?
We might consider extending the klp_shadow_* API to support the new livepatch tag.
Quoted 37 lines
Quoted 26 lines
Quoted 3 lines
That sounds like a viable solution. I'll look into it and see how we can implement it.
Does the following change look good to you ?
Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags
Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:
replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.
include/linux/livepatch.h +0
-0include/linux/livepatch.h struct klp_state { · line 128
Quoted 7 lines
include/linux/livepatch.h +3
-1@@ -123,7 +123,11 @@ struct klp_state {* @mod: reference to the live patch module * @objs: object entries for kernel objects to be patched * @states: system states that can get modified -* @replace: replace all actively used patches +* @replace: replace tag: +* = 0: Atomic replace is disabled; however, this patch remains +* eligible to be superseded by others. This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Quoted 1 line
Do we still need this category?
It can be superseded by any livepatch that has a non-zero tag set.
This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.
Quoted 14 lines
include/linux/livepatch.h struct klp_patch { · line 144
Quoted 12 lines
include/linux/livepatch.h +3
-1+* > 0: Atomic replace is enabled. Only existing patches with a +* matching replace tag will be superseded. * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries @@ -137,7 +141,7 @@ struct klp_patch {struct module *mod;struct klp_object *objs;struct klp_state *states; -bool replace; +unsigned int replace;This already breaks the backward compatibility
It doesn't break backward compatibility.
Quoted 8 lines
by changing the type
and semantic of this field. I would also change the name to better match the new semantic. What about using:* @replace_set: Livepatch using the same @replace_set will get atomically replaced, see also conflicts[*].
Quoted 8 lines
unsigned int
replace_set;[*] A livepatch module, livepatching an already livepatches function,
can be loaded only when it has the same @replace_set number.By other words, two livepatches conflict when they have a different @replace_set number and have at least one livepatched version in common.
Renaming it to @replace_set makes sense to me.
Quoted 38 lines
Quoted 28 lines
include/linux/livepatch.h +0
-0/* internal */ struct list_head list;kernel/livepatch/core.c +2
-0@@ -793,6 +793,8 @@ void klp_free_replaced_patches_async(struct klp_patch *new_patch) klp_for_each_patch_safe(old_patch, tmp_patch) {if (old_patch == new_patch)return ; +if (old_patch->replace && old_patch->replace != new_patch->replace) +continue ;klp_free_patch_async (old_patch);}}
kernel/livepatch/core.c +2
-0@@ -1194,6 +1196,8 @@ void klp_unpatch_replaced_patches(struct klp_patch *new_patch) klp_for_each_patch(old_patch) {if (old_patch == new_patch)return ; +if (old_patch->replace && old_patch->replace != new_patch->replace) +continue ;Quoted 2 lines
old_patch->enabled = false; klp_unpatch_objects(old_patch);This handles only the freeing part. More changes will be necessary:
+
klp_is_patch_compatible()must check also conflicts between livepatches with different @replace_set. The conflicts might be in the lists of:
Quoted 7 lines
+ livepatched functions
+ state IDs (aka callbacks and shadow variables IDs)+
klp_add_nops()must skip livepatches with another @replace_set+
klp_unpatch_replaced_patches()should unpatch only patches with the same @replace_set
I appreciate the guidance on this.
Quoted 8 lines
Finally, we would need to update existing selftests plus add new selftests.
It is possible that I have missed something.
Anyway, you should wait for more feedback before you do too much coding, especially the selftests are not needed at RFC stage.
Sure.
Signature · 2 lines
Regards Yafang
Petr Mladek
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 36 lines · Yafang Shao
Quoted 35 lines · Petr Mladek
Quoted 32 lines · Yafang Shao
Quoted 31 lines · Yafang Shao
Quoted 29 lines · Song Liu
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
Quoted 7 lines
Quoted 4 lines
I still think how to make the hybrid mode more secure:
+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?We might consider extending the klp_shadow_* API to support the new livepatch tag.
It would be nice to associate shadow variables with states so that we could check which shadow variables are used by each livepatch.
It is partially implemented in my earlier RFC, see
https://lore.kernel.org/all/[email protected]/
Quoted 36 lines
Quoted 36 lines
Quoted 25 lines
Quoted 2 lines
That sounds like a viable solution. I'll look into it and see how we can implement it.
Does the following change look good to you ?
Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags
Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:
replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.
include/linux/livepatch.h +0
-0include/linux/livepatch.h struct klp_state { · line 128
Quoted 7 lines
include/linux/livepatch.h +3
-1@@ -123,7 +123,11 @@ struct klp_state {* @mod: reference to the live patch module * @objs: object entries for kernel objects to be patched * @states: system states that can get modified -* @replace: replace all actively used patches +* @replace: replace tag: +* = 0: Atomic replace is disabled; however, this patch remains +* eligible to be superseded by others. This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Quoted 3 lines
Quoted 1 line
Do we still need this category?
It can be superseded by any livepatch that has a non-zero tag set.
And this exactly the weird thing.
A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.
Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.
Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.
So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.
Quoted 4 lines
This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.
IMHO, the backward compatibility would just create complexity and mess in this case.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
include/linux/livepatch.h +2
-0+* > 0: Atomic replace is enabled. Only existing patches with a +* matching replace tag will be superseded. * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries Quoted 8 lines
include/linux/livepatch.h struct klp_patch { · line 144
Quoted 6 lines
include/linux/livepatch.h +1
-1@@ -137,7 +141,7 @@ struct klp_patch {struct module *mod;struct klp_object *objs;struct klp_state *states; -bool replace; +unsigned int replace;This already breaks the backward compatibility
Quoted 1 line
It doesn't break backward compatibility.
It does. Livepatches with .replace flag set would need to define:
struct livepatch patch = {
.replace = <number>,
}instead of
struct livepatch patch = {
.replace = true,
}Best Regards,
Petr
Song Liu
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
On Wed, Apr 8, 2026 at 4:43 AM Petr Mladek <[email protected]> wrote:
[...]
Quoted 19 lines
Quoted 7 lines
Quoted 5 lines
This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Do we still need this category?
It can be superseded by any livepatch that has a non-zero tag set.
And this exactly the weird thing.
A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.
Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.
Quoted 8 lines
Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.
So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.
I think I see your point now. Existing code works as:
- replace=false doesn't replace anything
- replace=true replaces everything
If we assume false=0 and true=1, it is technically possible to define:
- replace_set=0 doesn't replace anything
- replace_set=1 replaces everything
- replace_set=2+ only replace the same replace_set
This is probably a little too complicated.
Quoted 7 lines
Quoted 4 lines
This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.
IMHO, the backward compatibility would just create complexity and mess in this case.
Given that livepatch is for expert users, I think we can make this work without backward compatibility. But breaking compatibility is always not preferred.
Thanks,
Song
Petr Mladek
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 30 lines · Song Liu
On Wed, Apr 8, 2026 at 4:43 AM Petr Mladek <[email protected]> wrote:
[...]Quoted 19 lines
Quoted 7 lines
Quoted 5 lines
This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Do we still need this category?
It can be superseded by any livepatch that has a non-zero tag set.
And this exactly the weird thing.
A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.
Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.
Quoted 8 lines
Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.
So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.
Quoted 8 lines
I think I see your point now. Existing code works as:
- replace=false doesn't replace anything
- replace=true replaces everythingIf we assume false=0 and true=1, it is technically possible to define:
- replace_set=0 doesn't replace anything
- replace_set=1 replaces everything
- replace_set=2+ only replace the samereplace_set
Yes. This well describes my point.
Quoted 7 lines
This is probably a little too complicated.
Quoted 5 lines
Quoted 4 lines
This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.
Quoted 6 lines
Quoted 2 lines
IMHO, the backward compatibility would just create complexity and mess in this case.
Given that livepatch is for expert users, I think we can make this work without backward compatibility. But breaking compatibility is always not preferred.
I believe that it is acceptable because:
1. It was always hard to combine no-replace and replace livepatches. I wonder if anyone combines them at all.
2. I believe that nobody tries to load the same livepatch module on different kernel versions. Instead, everyone prepares a custom livepatch module for each livepatched kernel version/release.
And the tooling for creating livepatches will need to be updated to use "number" instead of "true/false" anyway.
That said, it is easier to always use "0" for non-replace patches instead of assigning an unique "number" to avoid replacing. But I do not think that this would justify the complexity of having different semantic for 0, 1, and 2+ replace_set numbers.
Best Regards,
Petr
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
Quoted 53 lines · Petr Mladek
Quoted 30 lines · Song Liu
On Wed, Apr 8, 2026 at 4:43 AM Petr Mladek <[email protected]> wrote:
[...]Quoted 19 lines
Quoted 7 lines
Quoted 5 lines
This is weird semantic. Which livepatch tag would be allowed to supersede it, please?
Do we still need this category?
It can be superseded by any livepatch that has a non-zero tag set.
And this exactly the weird thing.
A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.
Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.
Quoted 8 lines
Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.
So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.
Quoted 8 lines
I think I see your point now. Existing code works as:
- replace=false doesn't replace anything
- replace=true replaces everythingIf we assume false=0 and true=1, it is technically possible to define:
- replace_set=0 doesn't replace anything
- replace_set=1 replaces everything
- replace_set=2+ only replace the samereplace_setYes. This well describes my point.
Quoted 10 lines
This is probably a little too complicated.
Quoted 7 lines
Quoted 4 lines
This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.
IMHO, the backward compatibility would just create complexity and mess in this case.
Quoted 8 lines
Quoted 3 lines
Given that livepatch is for expert users, I think we can make this work without backward compatibility. But breaking compatibility is always not preferred.
I believe that it is acceptable because:
1. It was always hard to combine no-replace and replace livepatches. I wonder if anyone combines them at all.
Because 'replace' patches can supersede 'no-replace' ones, users have to maintain a strict loading order. I doubt anyone actually combines them in production.
Quoted 4 lines
2. I believe that nobody tries to load the same livepatch module on different kernel versions. Instead, everyone prepares a custom livepatch module for each livepatched kernel version/release.
Correct. We always build and apply distinct livepatches for each specific kernel version.
Quoted 8 lines
And the tooling for creating livepatches will need to be updated to use "number" instead of "true/false" anyway.
That said, it is easier to always use "0" for non-replace patches instead of assigning an unique "number" to avoid replacing. But I do not think that this would justify the complexity of having different semantic for 0, 1, and 2+
replace_setnumbers.
Fair enough. Let's drop backward compatibility to keep the implementation simple.
--
Regards
Yafang
Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
In reply to Petr MladekQuoted 43 lines · Petr Mladek
Quoted 41 lines · Yafang Shao
Quoted 39 lines · Petr Mladek
Quoted 24 lines · Yafang Shao
Quoted 23 lines · Yafang Shao
Quoted 21 lines · Song Liu
On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in
klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.
We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.
I still think how to make the hybrid mode more secure:
+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?
Quoted 8 lines
Quoted 2 lines
We might consider extending the klp_shadow_* API to support the new livepatch tag.
It would be nice to associate shadow variables with states so that we could check which shadow variables are used by each livepatch.
It is partially implemented in my earlier RFC, see
https://lore.kernel.org/all/[email protected]/
This patch is still pending acceptance, but it offers a nice improvement. With your modifications, the remaining task would be to integrate a new replace_set into klp_state and update the API accordingly
[...]
Signature · 2 lines
Regards Yafang
Petr Mladek
maintainerRe: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch
In reply to Yafang ShaoQuoted 53 lines · Yafang Shao
Quoted 27 lines · Song Liu
On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]Quoted 16 lines
Quoted 4 lines
IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.
In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.
We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.
Quoted 5 lines
However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.
IIUC, the plan is:
Quoted 8 lines
- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?
That matches our expected use case.
Quoted 5 lines
I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade "off-band' livepatches, we will have to unload the old version and load the new version later.Right. That is how the non-atomic-replace patch works.
Quoted 2 lines
2. Any conflict with the off-band livepatches and regular livepatches will be difficult to manage.
We need to manage this conflict with a complex user script. That said, everything can be controlled from userspace.
Quoted 8 lines
Quoted 4 lines
IOW, we kind removed the benefit of cumulative
livepatches. For example, what shall we do if we really need two fixes to the same kernel functions: one from the original branch, the other from the off-band branch?We run tens of livepatches on our production servers and have never run into this issue. It's an extremely rare case — and if it does happen, a user script should be able to handle it just fine.
Could you please share the script? Or at least summarize the situations when this script detect a conflict and refuse loading a livepatch?
I believe that most/all of these checks can be implemented in the kernel. And if we agreed to add a hybrid mode than it should be added together with the checks.
We have already invested a lot of effort into make the kernel livepatching as safe as possible. From my POV, the most important parts are:
+ consistency model: Tasks are transitioned separately when they do not use any livepatched function.
+ atomic replace: Transition all livepatched functions at once.
If we agree to add the hybrid model then we should add it with some safety belts as well. And it would be nice to get inspiration about the safety checks from your script.
Best Regards,
Petr
RFC PATCH 4/4 livepatch: Implement livepatch hybrid mode
In reply to Yafang ShaoLivepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.
A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.
To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.
--- kernel/livepatch/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/livepatch/core.c +6 -0
@@ -665,6 +665,8 @@ static int klp_add_nops(struct klp_patch *patch)klp_for_each_object (old_patch, old_obj) {int err; +if (!old_patch->replaceable) +continue ; err =klp_add_object_nops (patch, old_obj);if (err)return err;
@@ -837,6 +839,8 @@ void klp_free_replaced_patches_async(struct klp_patch *new_patch)klp_for_each_patch_safe (old_patch, tmp_patch) {if (old_patch == new_patch)return ; +if (!old_patch->replaceable) +continue ;klp_free_patch_async (old_patch); } }
@@ -1239,6 +1243,8 @@ void klp_unpatch_replaced_patches(struct klp_patch *new_patch)if (old_patch == new_patch)return ; +if (!old_patch->replaceable) +continue ; old_patch->enabled = false;klp_unpatch_objects (old_patch); }
Signature · 1 line
2.47.3
Song Liu
maintainerHi Yafang,
Quoted 8 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
Quoted 7 lines
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.
Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.
Thanks,
Song
Quoted 17 lines · Song Liu
Hi Yafang,
Quoted 13 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
Quoted 9 lines
Quoted 2 lines
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.
Signature · 2 lines
Regards Yafang
Song Liu
maintainerQuoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu
Hi Yafang,
Quoted 13 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
Quoted 9 lines
Quoted 2 lines
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
Quoted 7 lines
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched
bond_xmit_3ad_xor_slave_get(), added a new hook calledbond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.
I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.
Can we use struct_ops instead of bpf_override_return for this case? This should make the solution more flexible.
Thanks,
Song
Quoted 43 lines · Song Liu
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu
Hi Yafang,
Quoted 13 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
Quoted 9 lines
Quoted 2 lines
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
Quoted 7 lines
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched
bond_xmit_3ad_xor_slave_get(), added a new hook calledbond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.
Quoted 3 lines
I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.
Right
Quoted 3 lines
Can we use
struct_opsinstead ofbpf_override_returnfor this case? This should make the solution more flexible.
Upstreaming struct_ops based BPF hooks is a challenging process, as seen in these examples:
https://lwn.net/Articles/1054030/
https://lwn.net/Articles/1043548/
Even when successful, upstreaming can take a significant amount of time—often longer than our production requirements allow. To bridge this gap, we developed this livepatch+BPF solution. This allows us to rapidly deploy new features without maintaining custom hooks in our local kernel. Because these livepatch-based hooks are lightweight, they minimize maintenance overhead and simplify kernel upgrades (e.g., from 6.1 to 6.18).
That said, we would still prefer to have our hooks accepted upstream to eliminate the need for self-maintenance entirely.
Signature · 2 lines
Regards Yafang
Song Liu
maintainerQuoted 61 lines · Yafang Shao
Quoted 43 lines · Song Liu
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu
Hi Yafang,
Quoted 13 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
Quoted 9 lines
Quoted 2 lines
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
Quoted 7 lines
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched
bond_xmit_3ad_xor_slave_get(), added a new hook calledbond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.Quoted 3 lines
I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.
Right
Quoted 3 lines
Can we use
struct_opsinstead ofbpf_override_returnfor this case? This should make the solution more flexible.Upstreaming
struct_opsbased BPF hooks is a challenging process, as seen in these examples:https://lwn.net/Articles/1054030/
https://lwn.net/Articles/1043548/
Quoted 7 lines
Even when successful, upstreaming can take a significant amount of time—often longer than our production requirements allow. To bridge this gap, we developed this livepatch+BPF solution. This allows us to rapidly deploy new features without maintaining custom hooks in our local kernel. Because these livepatch-based hooks are lightweight, they minimize maintenance overhead and simplify kernel upgrades (e.g., from 6.1 to 6.18).
I didn't mean upstream struct_ops.
We can define the struct_ops in an OOT kernel module. Then we can attach BPF programs to the struct_ops. We may need livepatch to connect the new struct_ops to original kernel logic.
I think kernel side of this solution is mostly available, but we may need some work on the toolchain side.
Does this make sense?
Thanks,
Song
Quoted 2 lines
That said, we would still prefer to have our hooks accepted upstream to eliminate the need for self-maintenance entirely.
Quoted 73 lines · Song Liu
Quoted 61 lines · Yafang Shao
Quoted 43 lines · Song Liu
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu
Hi Yafang,
Quoted 13 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
Quoted 9 lines
Quoted 2 lines
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
Quoted 7 lines
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched
bond_xmit_3ad_xor_slave_get(), added a new hook calledbond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.Quoted 3 lines
I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.
Right
Quoted 3 lines
Can we use
struct_opsinstead ofbpf_override_returnfor this case? This should make the solution more flexible.Upstreaming
struct_opsbased BPF hooks is a challenging process, as seen in these examples:https://lwn.net/Articles/1054030/
https://lwn.net/Articles/1043548/Quoted 7 lines
Even when successful, upstreaming can take a significant amount of time—often longer than our production requirements allow. To bridge this gap, we developed this livepatch+BPF solution. This allows us to rapidly deploy new features without maintaining custom hooks in our local kernel. Because these livepatch-based hooks are lightweight, they minimize maintenance overhead and simplify kernel upgrades (e.g., from 6.1 to 6.18).
I didn't mean upstream
struct_ops.
Quoted 8 lines
We can define the
struct_opsin an OOT kernel module. Then we can attach BPF programs to thestruct_ops. We may need livepatch to connect the newstruct_opsto original kernel logic.I think kernel side of this solution is mostly available, but we may need some work on the toolchain side.
Does this make sense?
Are there actual benefits to using struct_ops instead of bpf_override_return? So far, I’ve only found it adds complexity without much gain.
Can we add something like ALLOW_LIVEPATCH_ERROR_INJECTION() to allow error injection on functions defined inside a livepatch?
Signature · 2 lines
Regards Yafang
Song Liu
maintainerOn Mon, Apr 6, 2026 at 8:14 PM Yafang Shao <[email protected]> wrote:
[...]
Quoted 4 lines
Quoted 4 lines
We can define the
struct_opsin an OOT kernel module. Then we can attach BPF programs to thestruct_ops. We may need livepatch to connect the newstruct_opsto original kernel logic.
Quoted 8 lines
Quoted 4 lines
I think kernel side of this solution is mostly available, but we may need some work on the toolchain side.
Does this make sense?
Are there actual benefits to using
struct_opsinstead ofbpf_override_return? So far, I’ve only found it adds complexity without much gain.
Yes, struct_ops can be more powerful. For example, we can use something like the following to modify the content of /proc/cmdline with a bpf program. AFAICT, this is not possible with bpf_override_return.
Note that this example doesn't require kernel changes. This is actually a fun example. I will send the full code as a new self test.
Thanks,
Song
======= key logic of the livepatch module =======
static int livepatch_cmdline_proc_show(struct seq_file *m, void *v)
{
struct klp_bpf_cmdline_ops *ops = READ_ONCE(active_ops);
if (ops && ops->set_cmdline)
return ops->set_cmdline(m);
seq_printf(m, "%s: no struct_ops attached\n", THIS_MODULE->name);
return 0;
}static struct klp_func funcs[] = {
{
.old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show,
}, { }
};
========== key logic of the bpf program =========
SEC("struct_ops/set_cmdline")
int BPF_PROG(set_cmdline, struct seq_file *m)
{
char custom[] = "klp_bpf: custom cmdline\n";
bpf_klp_seq_write(m, custom, sizeof(custom) - 1);
return 0;
}SEC(".struct_ops.link")
struct klp_bpf_cmdline_ops cmdline_ops = {
.set_cmdline = (void *)set_cmdline,
};
Miroslav Benes
maintainerQuoted 2 lines
Can we add something like
ALLOW_LIVEPATCH_ERROR_INJECTION()to allow error injection on functions defined inside a livepatch?
No.
I am sorry but you always seem to find band aids to your set up and how you deal with live patches internally. While I can see that something like a hybrid mode might be useful to people if done right (and we are not there yet), the combination of it with bpf overrides or anything like that is not something I would like to see in upstream.
Miroslav
Quoted 11 lines · Miroslav Benes
Quoted 3 lines
Can we add something like
ALLOW_LIVEPATCH_ERROR_INJECTION()to allow error injection on functions defined inside a livepatch?No.
I am sorry but you always seem to find band aids to your set up and how you deal with live patches internally. While I can see that something like a hybrid mode might be useful to people if done right (and we are not there yet), the combination of it with bpf overrides or anything like that is not something I would like to see in upstream.
The upstream kernel already allows for combining BPF and livepatch to override functions. Song’s patch offers a great reference for implementing this without changing the kernel:
https://lore.kernel.org/bpf/[email protected]/
Signature · 2 lines
Regards Yafang
Quoted 2 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads.
Myabe it allows, or based on the rest of the mail not quite. But that is certainly not the intent at all, the intent is to fix critical bugs without downtime.
Quoted 6 lines
However, static livepatches lack
the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
Quoted 7 lines
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.
Whol f**. now. Is this a delayed April 1st post?
Quoted 18 lines · Christoph Hellwig
Quoted 2 lines · Yafang Shao
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads.
Myabe it allows, or based on the rest of the mail not quite. But that is certainly not the intent at all, the intent is to fix critical bugs without downtime.
Quoted 9 lines
However, static livepatches lack
the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.To address this, we propose a hybrid approach using BPF. Our production use case involves:
Quoted 6 lines
Quoted 4 lines
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.Whol f**. now. Is this a delayed April 1st post?
You're already in my spam list. Don't expect any further replies. Feel free to keep your verbose rubbish to yourself.
Signature · 2 lines
Regards Yafang
Masami Hiramatsu (Google)
maintainerHi Yafang,
Quoted 7 lines · On Thu, 2 Apr 2026 17:26:03 +0800 Yafang Shao <[email protected]> wrote:
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
Quoted 7 lines
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.
First of all, I don't like this approach to test a new feature in the kernel, because it sounds like allowing multiple different generations of implementations to coexist simultaneously. The standard kernel code is not designed to withstand such implementations.
For example, if you implement a well-designed framework in a specific subsystem, like Schedext, which allows multiple implementations extended with BPF to coexist, there's no problem (at least it's debatable).
But if it is for any function, it is dangerous feature. Bugs that occur in kernels that use this functionality cannot be addressed here. They need to be treated the same way as out-of-tree drivers or forked kernels. I mean, add a tainted flag for this feature. And we don't care of it.
Quoted 9 lines
A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.
To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.
Can you share your actual problem to be solved?
If the specific problem and the specific subsystem are clear, I think there is room to discuss it with the subsystem maintainer.
Quoted 6 lines
Furthermore, this mechanism provides a lower-maintenance alternative to out-of-tree BPF hooks. Given the complexities of upstreaming custom BPF hooks (e.g., [0], [1]), this hybrid mode allows for the maintenance of stable, minimal hook points via livepatching with significantly reduced maintenance burden.
Maintenance cost is the same. We need to add out-of-tree BPF hooks on source code. Maybe deploying cost will be reduced.
Thank you,
Quoted 21 lines
Link https://lwn.net/Articles/1054030/ [0]Link https://lwn.net/Articles/1043548/ [1]Yafang Shao (4):
trace: Simplify kprobe overridable function check
trace: Allow kprobes to override livepatched functions
livepatch: Add "replaceable" attribute toklp_patch
livepatch: Implement livepatch hybrid modeinclude/linux/livepatch.h | 2 ++ kernel/livepatch/core.c | 50 +++++++++++++++++++++++++++++++ kernel/trace/Kconfig | 14 +++++++++ kernel/trace/bpf_trace.c | 14 ++++++--- kernel/trace/trace_kprobe.c | 49 ++++++++++++------------------ kernel/trace/trace_probe.h | 59 +++++++++++++++++++++++++++---------- 6 files changed, 139 insertions(+), 49 deletions(-)Signature · 1 line
2.47.3
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 18 lines · Masami Hiramatsu
Hi Yafang,
Quoted 12 lines · On Thu, 2 Apr 2026 17:26:03 +0800 Yafang Shao <[email protected]> wrote:
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
Quoted 7 lines
Quoted 2 lines
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.First of all, I don't like this approach to test a new feature in the kernel, because it sounds like allowing multiple different generations of implementations to coexist simultaneously. The standard kernel code is not designed to withstand such implementations.
However, this approach is invaluable for rapidly deploying new kernel features to production servers without downtime. Upgrading kernels across a large fleet remains a significant challenge.
Quoted 9 lines
For example, if you implement a well-designed framework in a specific subsystem, like Schedext, which allows multiple implementations extended with BPF to coexist, there's no problem (at least it's debatable).
But if it is for any function, it is dangerous feature. Bugs that occur in kernels that use this functionality cannot be addressed here. They need to be treated the same way as out-of-tree drivers or forked kernels. I mean, add a tainted flag for this feature. And we don't care of it.
Agreed. This should be handled as an OOT module rather than part of the core kernel.
Quoted 7 lines
Quoted 7 lines
A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.
Quoted 5 lines
Quoted 3 lines
To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.
Can you share your actual problem to be solved?
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.
[...]
Signature · 2 lines
Regards Yafang
Masami Hiramatsu (Google)
maintainerQuoted 22 lines · On Sun, 12 Apr 2026 21:50:31 +0800 Yafang Shao <[email protected]> wrote:
Quoted 21 lines · Masami Hiramatsu
Hi Yafang,
Quoted 14 lines · On Thu, 2 Apr 2026 17:26:03 +0800 Yafang Shao <[email protected]> wrote:
Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.
To address this, we propose a hybrid approach using BPF. Our production use case involves:
1. Deploying a Livepatch function to serve as a stable BPF hook.
2. Utilizing
bpf_override_return()to dynamically modify the return value of that hook based on the current task's context.
Quoted 8 lines
Quoted 4 lines
First of all, I don't like this approach to test a new feature in the kernel, because it sounds like allowing multiple different generations of implementations to coexist simultaneously. The standard kernel code is not designed to withstand such implementations.
However, this approach is invaluable for rapidly deploying new kernel features to production servers without downtime. Upgrading kernels across a large fleet remains a significant challenge.
I think that downtime should be accepted as a cost for stability in general. If your new kernel feature has a bug and causes a crash, anyway it gets your servers down.
Quoted 30 lines
Quoted 9 lines
For example, if you implement a well-designed framework in a specific subsystem, like Schedext, which allows multiple implementations extended with BPF to coexist, there's no problem (at least it's debatable).
But if it is for any function, it is dangerous feature. Bugs that occur in kernels that use this functionality cannot be addressed here. They need to be treated the same way as out-of-tree drivers or forked kernels. I mean, add a tainted flag for this feature. And we don't care of it.
Agreed. This should be handled as an OOT module rather than part of the core kernel.
Quoted 7 lines
Quoted 7 lines
A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.
Quoted 5 lines
Quoted 3 lines
To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.
Can you share your actual problem to be solved?
Here is an example we recently deployed on our production servers:
https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
Quoted 7 lines
In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched
bond_xmit_3ad_xor_slave_get(), added a new hook calledbond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.
In this case, you can make specific livepatch or kernel module to replace the kernel function without using BPF on your server.
The BGP trafic in bonding device seems very specific, so it may not cause a trouble, but this is very generic change, which allows user to change more core kernel feature, e.g. memory management or scheduler etc.
Excessive degrees of freedom introduce uncertainty and instability into a system. While the functionality is interesting, it would be a way to generalize schedext in an uncontrolled way.
At a minimum, some form of build time and runtime constraint, along with a taint flag that clearly indicates in the crash logs that this feature is being used, would be necessary. (It means this is should not be used in production environment.)
Thank you,
Quoted 6 lines
[...]
Signature · 2 lines
Regards Yafang
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>