Skip to contentSkip to conversation

trace, livepatch: Allow kprobe return overriding for livepatched functions

RFC PATCH NAK Yafang Shao 11 participants 49 messages Apr 2 – Apr 15

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover trace, livepatch: Allow kprobe return overriding for livepatched functions

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.

To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.

Furthermore, this mechanism provides a lower-maintenance alternative to out-of-tree BPF hooks. Given the complexities of upstreaming custom BPF hooks (e.g., [0], [1]), this hybrid mode allows for the maintenance of stable, minimal hook points via livepatching with significantly reduced maintenance burden.

Yafang Shao (4):
trace: Simplify kprobe overridable function check
trace: Allow kprobes to override livepatched functions
livepatch: Add "replaceable" attribute to klp_patch
livepatch: Implement livepatch hybrid mode

 include/linux/livepatch.h   |  2 ++
 kernel/livepatch/core.c     | 50 +++++++++++++++++++++++++++++++
 kernel/trace/Kconfig        | 14 +++++++++
 kernel/trace/bpf_trace.c    | 14 ++++++---
 kernel/trace/trace_kprobe.c | 49 ++++++++++++------------------
 kernel/trace/trace_probe.h  | 59 +++++++++++++++++++++++++++----------
 6 files changed, 139 insertions(+), 49 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

RFC PATCH 1/4 trace: Simplify kprobe overridable function check

Simplify the logic for checking overridable kprobe functions by removing redundant code.

No functional change.

---
 kernel/trace/bpf_trace.c    | 13 ++++++---
 kernel/trace/trace_kprobe.c | 40 +++++----------------------
 kernel/trace/trace_probe.h  | 54 ++++++++++++++++++++++++++-----------
 3 files changed, 54 insertions(+), 53 deletions(-)
kernel/trace/bpf_trace.c +9 -4
index 0b040a417442..c901ace836cb 100644
@@ -1929,10 +1929,15 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
 	 * Kprobe override only works if they are on the function entry,
 	 * and only if they are on the opt-in list.
 	 */
-	if (prog->kprobe_override &&
-	    (!trace_kprobe_on_func_entry(event->tp_event) ||
-	     !trace_kprobe_error_injectable(event->tp_event)))
-		return -EINVAL;
+	if (prog->kprobe_override) {
+		struct trace_kprobe *tp = trace_kprobe_primary_from_call(event->tp_event);
+
+		if (!tp)
+			return -EINVAL;
+		if (!trace_kprobe_on_func_entry(tp) ||
+		    !trace_kprobe_error_injectable(tp))
+			return -EINVAL;
+	}
 
 	mutex_lock(&bpf_event_mutex);
 
kernel/trace/trace_kprobe.c +6 -34
index a5dbb72528e0..768702674a5c 100644
@@ -53,17 +53,6 @@ static struct dyn_event_operations trace_kprobe_ops = {
 	.match = trace_kprobe_match,
 };
 
-/*
- * Kprobe event core functions
- */
-struct trace_kprobe {
-	struct dyn_event	devent;
-	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
-	unsigned long __percpu *nhit;
-	const char		*symbol;	/* symbol name */
-	struct trace_probe	tp;
-};
-
 static bool is_trace_kprobe(struct dyn_event *ev)
 {
 	return ev->ops == &trace_kprobe_ops;
@@ -212,33 +201,16 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 	return addr;
 }
 
-static nokprobe_inline struct trace_kprobe *
-trace_kprobe_primary_from_call(struct trace_event_call *call)
-{
-	struct trace_probe *tp;
-
-	tp = trace_probe_primary_from_call(call);
-	if (WARN_ON_ONCE(!tp))
-		return NULL;
-
-	return container_of(tp, struct trace_kprobe, tp);
-}
-
-bool trace_kprobe_on_func_entry(struct trace_event_call *call)
+bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
 {
-	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
-
-	return tk ? (kprobe_on_func_entry(tk->rp.kp.addr,
-			tk->rp.kp.addr ? NULL : tk->rp.kp.symbol_name,
-			tk->rp.kp.addr ? 0 : tk->rp.kp.offset) == 0) : false;
+	return !kprobe_on_func_entry(tp->rp.kp.addr,
+			tp->rp.kp.addr ? NULL : tp->rp.kp.symbol_name,
+			tp->rp.kp.addr ? 0 : tp->rp.kp.offset);
 }
 
-bool trace_kprobe_error_injectable(struct trace_event_call *call)
+bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 {
-	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
-
-	return tk ? within_error_injection_list(trace_kprobe_address(tk)) :
-	       false;
+	return within_error_injection_list(trace_kprobe_address(tp));
 }
 
 static int register_kprobe_event(struct trace_kprobe *tk);
kernel/trace/trace_probe.h +39 -15

78 lines not on this page · raw message

Signature · 1 line
2.47.3
#2 lore raw

Re: [RFC PATCH 1/4] trace: Simplify kprobe overridable function check

Quoted 4 lines · On Thu, 2 Apr 2026 17:26:04 +0800 Yafang Shao <[email protected]> wrote:

Simplify the logic for checking overridable kprobe functions by removing redundant code.

No functional change.

NACK.

trace_kprobe must be hidden inside the trace_kprobe.c. It is not designed to be exposed.

Thank you,

Quoted 183 lines
---
 kernel/trace/bpf_trace.c    | 13 ++++++---
 kernel/trace/trace_kprobe.c | 40 +++++----------------------
 kernel/trace/trace_probe.h  | 54 ++++++++++++++++++++++++++-----------
 3 files changed, 54 insertions(+), 53 deletions(-)
kernel/trace/bpf_trace.c +9 -4
index 0b040a417442..c901ace836cb 100644
@@ -1929,10 +1929,15 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
 	 * Kprobe override only works if they are on the function entry,
 	 * and only if they are on the opt-in list.
 	 */
-	if (prog->kprobe_override &&
-	    (!trace_kprobe_on_func_entry(event->tp_event) ||
-	     !trace_kprobe_error_injectable(event->tp_event)))
-		return -EINVAL;
+	if (prog->kprobe_override) {
+		struct trace_kprobe *tp = trace_kprobe_primary_from_call(event->tp_event);
+
+		if (!tp)
+			return -EINVAL;
+		if (!trace_kprobe_on_func_entry(tp) ||
+		    !trace_kprobe_error_injectable(tp))
+			return -EINVAL;
+	}
 
 	mutex_lock(&bpf_event_mutex);
 
kernel/trace/trace_kprobe.c +6 -34
index a5dbb72528e0..768702674a5c 100644
@@ -53,17 +53,6 @@ static struct dyn_event_operations trace_kprobe_ops = {
 	.match = trace_kprobe_match,
 };
 
-/*
- * Kprobe event core functions
- */
-struct trace_kprobe {
-	struct dyn_event	devent;
-	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
-	unsigned long __percpu *nhit;
-	const char		*symbol;	/* symbol name */
-	struct trace_probe	tp;
-};
-
 static bool is_trace_kprobe(struct dyn_event *ev)
 {
 	return ev->ops == &trace_kprobe_ops;
@@ -212,33 +201,16 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 	return addr;
 }
 
-static nokprobe_inline struct trace_kprobe *
-trace_kprobe_primary_from_call(struct trace_event_call *call)
-{
-	struct trace_probe *tp;
-
-	tp = trace_probe_primary_from_call(call);
-	if (WARN_ON_ONCE(!tp))
-		return NULL;
-
-	return container_of(tp, struct trace_kprobe, tp);
-}
-
-bool trace_kprobe_on_func_entry(struct trace_event_call *call)
+bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
 {
-	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
-
-	return tk ? (kprobe_on_func_entry(tk->rp.kp.addr,
-			tk->rp.kp.addr ? NULL : tk->rp.kp.symbol_name,
-			tk->rp.kp.addr ? 0 : tk->rp.kp.offset) == 0) : false;
+	return !kprobe_on_func_entry(tp->rp.kp.addr,
+			tp->rp.kp.addr ? NULL : tp->rp.kp.symbol_name,
+			tp->rp.kp.addr ? 0 : tp->rp.kp.offset);
 }
 
-bool trace_kprobe_error_injectable(struct trace_event_call *call)
+bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 {
-	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
-
-	return tk ? within_error_injection_list(trace_kprobe_address(tk)) :
-	       false;
+	return within_error_injection_list(trace_kprobe_address(tp));
 }
 
 static int register_kprobe_event(struct trace_kprobe *tk);
kernel/trace/trace_probe.h +39 -15
index 9fc56c937130..958eb78a9068 100644
@@ -30,6 +30,7 @@
 
 #include "trace.h"
 #include "trace_output.h"
+#include "trace_dynevent.h"
 
 #define MAX_TRACE_ARGS		128
 #define MAX_ARGSTR_LEN		63
@@ -210,21 +211,6 @@ DECLARE_BASIC_PRINT_TYPE_FUNC(symbol);
 #define ASSIGN_FETCH_TYPE_END {}
 #define MAX_ARRAY_LEN	64
 
-#ifdef CONFIG_KPROBE_EVENTS
-bool trace_kprobe_on_func_entry(struct trace_event_call *call);
-bool trace_kprobe_error_injectable(struct trace_event_call *call);
-#else
-static inline bool trace_kprobe_on_func_entry(struct trace_event_call *call)
-{
-	return false;
-}
-
-static inline bool trace_kprobe_error_injectable(struct trace_event_call *call)
-{
-	return false;
-}
-#endif /* CONFIG_KPROBE_EVENTS */
-
 struct probe_arg {
 	struct fetch_insn	*code;
 	bool			dynamic;/* Dynamic array (string) is used */
@@ -271,6 +257,32 @@ struct event_file_link {
 	struct list_head		list;
 };
 
+/*
+ * Kprobe event core functions
+ */
+struct trace_kprobe {
+	struct dyn_event	devent;
+	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
+	unsigned long __percpu	*nhit;
+	const char		*symbol;	/* symbol name */
+	struct trace_probe	tp;
+};
+
+#ifdef CONFIG_KPROBE_EVENTS
+bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
+bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
+#else
+static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
+{
+	return false;
+}
+
+static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
+{
+	return false;
+}
+#endif /* CONFIG_KPROBE_EVENTS */
+
 static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
 {
 	return smp_load_acquire(&tp->event->flags);
@@ -329,6 +341,18 @@ trace_probe_primary_from_call(struct trace_event_call *call)
 	return list_first_entry_or_null(&tpe->probes, struct trace_probe, list);
 }
 
+static nokprobe_inline struct trace_kprobe *
+trace_kprobe_primary_from_call(struct trace_event_call *call)
+{
+	struct trace_probe *tp;
+
+	tp = trace_probe_primary_from_call(call);
+	if (WARN_ON_ONCE(!tp))
+		return NULL;
+
+	return container_of(tp, struct trace_kprobe, tp);
+}
+
 static inline struct list_head *trace_probe_probe_list(struct trace_probe *tp)
 {
 	return &tp->event->probes;
Signature · 1 line
2.47.3
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#3 lore raw

RFC PATCH 2/4 trace: Allow kprobes to override livepatched functions

In reply to Yafang Shao

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

---
 kernel/trace/Kconfig        | 14 ++++++++++++++
 kernel/trace/bpf_trace.c    |  3 ++-
 kernel/trace/trace_kprobe.c | 17 +++++++++++++++++
 kernel/trace/trace_probe.h  |  5 +++++
 4 files changed, 38 insertions(+), 1 deletion(-)
kernel/trace/Kconfig +14 -0
index 49de13cae428..db712c8cb745 100644
@@ -1279,6 +1279,20 @@ config HIST_TRIGGERS_DEBUG
 
           If unsure, say N.
 
+config KPROBE_OVERRIDE_KLP_FUNC
+	bool "Allow kprobes to override livepatched functions"
+	depends on KPROBES && LIVEPATCH
+	help
+	  This option allows BPF programs to use kprobes to override functions
+	  that have already been patched by Livepatch (KLP).
+
+	  Enabling this provides a mechanism to dynamically control execution
+	  flow without requiring a reboot or a new livepatch module. It
+	  effectively combines the persistence of livepatching with the
+	  programmability of BPF.
+
+	  If unsure, say N.
+
 source "kernel/trace/rv/Kconfig"
 
 endif # FTRACE
kernel/trace/bpf_trace.c +2 -1
index c901ace836cb..08ae2b1a912c 100644
@@ -1935,7 +1935,8 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
 		if (!tp)
 			return -EINVAL;
 		if (!trace_kprobe_on_func_entry(tp) ||
-		    !trace_kprobe_error_injectable(tp))
+		    (!trace_kprobe_error_injectable(tp) &&
+		     !trace_kprobe_klp_func_overridable(tp)))
 			return -EINVAL;
 	}
 
kernel/trace/trace_kprobe.c +17 -0
index 768702674a5c..6f05451fbc76 100644
@@ -213,6 +213,23 @@ bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 	return within_error_injection_list(trace_kprobe_address(tp));
 }
 
+bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
+{
+	bool overridable = false;
+#ifdef CONFIG_KPROBE_OVERRIDE_KLP_FUNC
+	struct module *mod;
+	unsigned long addr;
+
+	addr = trace_kprobe_address(tp);
+	rcu_read_lock();
+	mod = __module_address(addr);
+	if (mod && mod->klp)
+		overridable = true;
+	rcu_read_unlock();
+#endif
+	return overridable;
+}
+
 static int register_kprobe_event(struct trace_kprobe *tk);
 static int unregister_kprobe_event(struct trace_kprobe *tk);
 
kernel/trace/trace_probe.h +5 -0
index 958eb78a9068..84bd2617db7c 100644
@@ -271,6 +271,7 @@ struct trace_kprobe {
 #ifdef CONFIG_KPROBE_EVENTS
 bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
 bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
+bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp);
 #else
 static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
 {
@@ -281,6 +282,10 @@ static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 {
 	return false;
 }
+static inline bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
+{
+	return false;
+}
 #endif /* CONFIG_KPROBE_EVENTS */
 
 static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
Signature · 1 line
2.47.3
#4 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 3 lines · Yafang Shao

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

Hi, Yafang. This is a interesting idea.

For now, the bpf_override_return() can only be used on the kernel functions that allow error injection to prevent the BPF program from crash the kernel. If we use it on the kernel functions that patched by the KLP, we can crash the kernel easily by return a invalid value with bpf_override_return(), right? (Of course, we can crash the kernel easily with KLP too ;)

I haven't figure out the use case yet. Can KLP be used together with the BPF program that use bpf_override_return()? The KLP will modify the RIP on the stack, and the bpf_override_return() will modify it too. AFAIK, there can't be two ftrace_ops that both have the FTRACE_OPS_FL_IPMODIFY flag. Did I miss something?

It will be helpful for me to understand the use case if a selftests is offered :)

BTW, if we allow the usage of bpf_override_return() on the KLP patched function, we should allow the usage of BPF_MODIFY_RETURN on this case too, right?

Thanks!
Menglong Dong

Quoted 100 lines
---
 kernel/trace/Kconfig        | 14 ++++++++++++++
 kernel/trace/bpf_trace.c    |  3 ++-
 kernel/trace/trace_kprobe.c | 17 +++++++++++++++++
 kernel/trace/trace_probe.h  |  5 +++++
 4 files changed, 38 insertions(+), 1 deletion(-)
kernel/trace/Kconfig +14 -0
index 49de13cae428..db712c8cb745 100644
@@ -1279,6 +1279,20 @@ config HIST_TRIGGERS_DEBUG
 
           If unsure, say N.
 
+config KPROBE_OVERRIDE_KLP_FUNC
+	bool "Allow kprobes to override livepatched functions"
+	depends on KPROBES && LIVEPATCH
+	help
+	  This option allows BPF programs to use kprobes to override functions
+	  that have already been patched by Livepatch (KLP).
+
+	  Enabling this provides a mechanism to dynamically control execution
+	  flow without requiring a reboot or a new livepatch module. It
+	  effectively combines the persistence of livepatching with the
+	  programmability of BPF.
+
+	  If unsure, say N.
+
 source "kernel/trace/rv/Kconfig"
 
 endif # FTRACE
kernel/trace/bpf_trace.c +2 -1
index c901ace836cb..08ae2b1a912c 100644
@@ -1935,7 +1935,8 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
 		if (!tp)
 			return -EINVAL;
 		if (!trace_kprobe_on_func_entry(tp) ||
-		    !trace_kprobe_error_injectable(tp))
+		    (!trace_kprobe_error_injectable(tp) &&
+		     !trace_kprobe_klp_func_overridable(tp)))
 			return -EINVAL;
 	}
 
kernel/trace/trace_kprobe.c +17 -0
index 768702674a5c..6f05451fbc76 100644
@@ -213,6 +213,23 @@ bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 	return within_error_injection_list(trace_kprobe_address(tp));
 }
 
+bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
+{
+	bool overridable = false;
+#ifdef CONFIG_KPROBE_OVERRIDE_KLP_FUNC
+	struct module *mod;
+	unsigned long addr;
+
+	addr = trace_kprobe_address(tp);
+	rcu_read_lock();
+	mod = __module_address(addr);
+	if (mod && mod->klp)
+		overridable = true;
+	rcu_read_unlock();
+#endif
+	return overridable;
+}
+
 static int register_kprobe_event(struct trace_kprobe *tk);
 static int unregister_kprobe_event(struct trace_kprobe *tk);
 
kernel/trace/trace_probe.h +5 -0
index 958eb78a9068..84bd2617db7c 100644
@@ -271,6 +271,7 @@ struct trace_kprobe {
 #ifdef CONFIG_KPROBE_EVENTS
 bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
 bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
+bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp);
 #else
 static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
 {
@@ -281,6 +282,10 @@ static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 {
 	return false;
 }
+static inline bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
+{
+	return false;
+}
 #endif /* CONFIG_KPROBE_EVENTS */
 
 static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
#5 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 6 lines · Menglong Dong
Quoted 3 lines · Yafang Shao

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

Quoted 8 lines

Hi, Yafang. This is a interesting idea.

For now, the bpf_override_return() can only be used on the kernel functions that allow error injection to prevent the BPF program from crash the kernel. If we use it on the kernel functions that patched by the KLP, we can crash the kernel easily by return a invalid value with bpf_override_return(), right? (Of course, we can crash the kernel easily with KLP too ;)

Right.
Livepatch already grants the power to modify the kernel at will; allowing BPF to override a patched function simply adds a layer of runtime programmability to an existing modification.

Quoted 3 lines

I haven't figure out the use case yet. Can KLP be used together with the BPF program that use bpf_override_return()?

The two mechanisms do not target the same entry point: whileKLP modifies the original kernel function, bpf_override_return() is applied to the newly patched function provided by the KLP module.

Quoted 4 lines

The KLP will modify
the RIP on the stack, and the bpf_override_return() will modify it too. AFAIK, there can't be two ftrace_ops that both have the FTRACE_OPS_FL_IPMODIFY flag. Did I miss something?

Correct, but as noted, they target different functions

Quoted 3 lines

It will be helpful for me to understand the use case if a selftests is offered :)

Here is a recent use case from our production environment.

- The livepatch

drivers/net/bonding/bond_main.c +0 -0
index e378bbe5705f..047e937bfa6d 100644
@@ -5175,12 +5175,22 @@ int bond_update_slave_arr(struct bonding

*bond, struct slave *skipslave)

        return ret;

}

+/* noclone to avoid bond_get_slave_hook.constprop.0 */ +__attribute__((__noclone__, __noinline__))
+int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count) +{
+ return -1;
+}

static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
struct sk_buff *skb, struct bond_up_slave *slaves)

 {
        struct slave *slave;
        unsigned int count;
+       int slave_idx;
        u32 hash;

        hash = bond_xmit_hash(bond, skb);
@@ -5188,6 +5198,13 @@ static struct slave
*bond_xmit_3ad_xor_slave_get(struct bonding *bond,
        if (unlikely(!count))
                return NULL;
+       /* Try BPF hook first - returns slave index directly */
+       slave_idx = bond_get_slave_hook(skb, hash, count);
+       /* If BPF hook returned valid slave index, use it */
+       if (slave_idx >= 0 && slave_idx < count) {
+               slave = slaves->arr[slave_idx];
+               return slave;
+       }
        slave = slaves->arr[hash % count];
        return slave;

}

- The BPF program

SEC("kprobe/bond_get_slave_hook")
int BPF_KPROBE(slave_selector, struct sk_buff *skb, u32 hash, u32 count)

{
        unsigned short net_hdr_off;
        unsigned char *head;
        struct iphdr iph;
        int *slave_idx;
        __u32 daddr;

        __u16 proto = BPF_CORE_READ(skb, protocol);
        if (proto != bpf_htons(0x0800))
                return 0;

        head = BPF_CORE_READ(skb, head);
        net_hdr_off = BPF_CORE_READ(skb, network_header);

        if (bpf_probe_read_kernel(&iph, sizeof(iph), head + net_hdr_off) != 0)
                return 0;

        daddr = iph.daddr;
        slave_idx = bpf_map_lookup_elem(&ip_slave_map, &daddr);
        if (slave_idx) {
                int idx = *slave_idx;

                if (idx >= 0 && idx < (int)count)
                        bpf_override_return(ctx, idx);
        }
        return 0;
}
Quoted 4 lines

BTW, if we allow the usage of bpf_override_return() on the KLP patched function, we should allow the usage of BPF_MODIFY_RETURN on this case too, right?

It's a possibility, but I haven't tested that specifically yet.

Signature · 2 lines
Regards
Yafang
#6 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

On 2026/4/2 21:20 Yafang Shao <[email protected]> write:

Quoted 9 lines · Menglong Dong
Quoted 8 lines · Yafang Shao
Quoted 3 lines

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

Hi, Yafang. This is a interesting idea.

[...]

Review

Quoted 7 lines
(quoted diff) +6 -0
+/* noclone to avoid bond_get_slave_hook.constprop.0 */
+__attribute__((__noclone__, __noinline__))
+int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count)
+{
+       return -1;
+}

Hi, yafang.

I see what you mean now. So you want to allow BPF program override the return of all the kernel functions in a KLP module.

I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of bpf_override_write().

What's more, this is a little weird for me. If we allow to use bpf_override_return() for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?

Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in your example? Then we can override its return directly. This is a more reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.

WDYT?

BTW, this is a BPF modification, so maybe we can use "bpf: xxx" for the title of this patch. Then, the BPF maintainers can notice this patch ;)

Thanks!
Menglong Dong

Quoted 69 lines

static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
struct sk_buff *skb, struct bond_up_slave *slaves)

 {
        struct slave *slave;
        unsigned int count;
(quoted diff) +8 -0
+       int slave_idx;
        u32 hash;
 
        hash = bond_xmit_hash(bond, skb);
@@ -5188,6 +5198,13 @@ static struct slave
 *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
        if (unlikely(!count))
                return NULL;
 
+       /* Try BPF hook first - returns slave index directly */
+       slave_idx = bond_get_slave_hook(skb, hash, count);
+       /* If BPF hook returned valid slave index, use it */
+       if (slave_idx >= 0 && slave_idx < count) {
+               slave = slaves->arr[slave_idx];
+               return slave;
+       }
        slave = slaves->arr[hash % count];
        return slave;

}

- The BPF program

SEC("kprobe/bond_get_slave_hook")
int BPF_KPROBE(slave_selector, struct sk_buff *skb, u32 hash, u32 count)

{
        unsigned short net_hdr_off;
        unsigned char *head;
        struct iphdr iph;
        int *slave_idx;
        __u32 daddr;

        __u16 proto = BPF_CORE_READ(skb, protocol);
        if (proto != bpf_htons(0x0800))
                return 0;

        head = BPF_CORE_READ(skb, head);
        net_hdr_off = BPF_CORE_READ(skb, network_header);

        if (bpf_probe_read_kernel(&iph, sizeof(iph), head + net_hdr_off) != 0)
                return 0;

        daddr = iph.daddr;
        slave_idx = bpf_map_lookup_elem(&ip_slave_map, &daddr);
        if (slave_idx) {
                int idx = *slave_idx;

                if (idx >= 0 && idx < (int)count)
                        bpf_override_return(ctx, idx);
        }
        return 0;
}
Quoted 4 lines

BTW, if we allow the usage of bpf_override_return() on the KLP patched function, we should allow the usage of BPF_MODIFY_RETURN on this case too, right?

It's a possibility, but I haven't tested that specifically yet.

Signature · 2 lines
Regards
Yafang
maintainer
#7 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 3 lines · On Fri, 03 Apr 2026 18:25:59 +0800 Menglong Dong <[email protected]>…

I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of bpf_override_write().

Right, livepatching may allow for rapid experimentation but that is not its purpose. It is for fixing production systems without having to reboot. Using BPF to change the return of a function is a huge security issue.

Quoted 5 lines

What's more, this is a little weird for me. If we allow to use bpf_override_return() for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?

Right.

Quoted 5 lines

Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in your example? Then we can override its return directly. This is a more reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.

If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.

The command line should be something like:

allow_bpf_to_rootkit_functions

-- Steve

#8 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 26 lines · Steven Rostedt
Quoted 3 lines · On Fri, 03 Apr 2026 18:25:59 +0800 Menglong Dong <[email protected]>…

I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of bpf_override_write().

Right, livepatching may allow for rapid experimentation but that is not its purpose. It is for fixing production systems without having to reboot. Using BPF to change the return of a function is a huge security issue.

Quoted 5 lines

What's more, this is a little weird for me. If we allow to use bpf_override_return() for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?

Right.

Quoted 5 lines

Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in your example? Then we can override its return directly. This is a more reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.

Quoted 8 lines

If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.

The command line should be something like:

allow_bpf_to_rootkit_functions

The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In the next revision, I will rename this to CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel command-line parameter, allow_bpf_to_rootkit_functions, to control it.

--
Regards

Yafang

#9 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 9 lines · Yafang Shao
Quoted 9 lines

If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.

The command line should be something like:

Quoted 7 lines
Quoted 1 line

allow_bpf_to_rootkit_functions

The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In the next revision, I will rename this to CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel command-line parameter, allow_bpf_to_rootkit_functions, to control it.

No. Even with extra config this is not ok.

#10 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 13 lines · Alexei Starovoitov
Quoted 11 lines · Yafang Shao
Quoted 10 lines

If this were to go in, I say it would require both a kernel config, with a big warning about this being a security hole, and a kernel command line option to enable it, so that people don't accidentally have it enabled in their config.

The command line should be something like:

allow_bpf_to_rootkit_functions

Quoted 7 lines
Quoted 5 lines

The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In the next revision, I will rename this to CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel command-line parameter, allow_bpf_to_rootkit_functions, to control it.

No. Even with extra config this is not ok.

I will send patch #3 and #4 as a standalone patchset to upstream the hybrid livepatch first and then figure out how to move allow_bpf_to_rootkit_functions forward.

--
Regards
Yafang

#11 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

In reply to Menglong Dong
Quoted 22 lines · Menglong Dong

On 2026/4/2 21:20 Yafang Shao <[email protected]> write:

Quoted 9 lines · Menglong Dong
Quoted 8 lines · Yafang Shao
Quoted 3 lines

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

Hi, Yafang. This is a interesting idea.

[...]

Review

Quoted 7 lines
(quoted diff) +6 -0
+/* noclone to avoid bond_get_slave_hook.constprop.0 */
+__attribute__((__noclone__, __noinline__))
+int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count)
+{
+       return -1;
+}

Hi, yafang.

Quoted 6 lines

I see what you mean now. So you want to allow BPF program override the return of all the kernel functions in a KLP module.

I think the security problem is a big issue. Image that we have a KLP in our environment. Any users can crash the kernel by hook a BPF program on it with the calling of bpf_override_write().

This feature is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option, which is disabled by default. Consequently, the user must explicitly enable this option to utilize the feature.

Quoted 5 lines

What's more, this is a little weird for me. If we allow to use bpf_override_return() for the kernel functions in a KLP, why not we allow it in a common kernel module, as KLP is a kind of kernel module. Then, why not we allow to use it for all the kernel functions?

By leveraging KLP, we can rapidly deploy new features without interrupting production workloads. Accordingly, this feature is specifically targeted at KLP-patched functions to maintain that seamless delivery model.

Quoted 5 lines

Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in your example? Then we can override its return directly. This is a more reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that anyone can modify the return of this function safely.

It is unfortunate that ALLOW_ERROR_INJECTION() is incompatible with KLP-patched functions, as this limits our ability to perform fault injection on livepatched code

Quoted 5 lines

WDYT?

BTW, this is a BPF modification, so maybe we can use "bpf: xxx" for the title of this patch. Then, the BPF maintainers can notice this patch ;)

I agree with the suggestion. I will update the subject prefix to "trace, bpf:" in the next version.

Signature · 2 lines
Regards
Yafang
maintainer
#12 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

In reply to Yafang Shao

Hi,

Quoted 3 lines · Yafang Shao

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

this is imprecise if I read the code correctly. You want to override live patch functions, not the original ones which are live patched.

I also think that if nothing else, it needs to be more specific then just checking mod->klp. It should check if a function itself in klp module is overridable to keep it as limited as possible. Even with that, the concerns expressed by the others still apply.

---
Miroslav

#13 lore raw

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

Quoted 11 lines · Miroslav Benes

Hi,

Quoted 3 lines · Yafang Shao

Introduce the ability for kprobes to override the return values of functions that have been livepatched. This functionality is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

this is imprecise if I read the code correctly. You want to override live patch functions, not the original ones which are live patched.

Correct. The BPF program will override the livepatched functions rather than the original ones.

Quoted 4 lines

I also think that if nothing else, it needs to be more specific then just checking mod->klp. It should check if a function itself in klp module is overridable to keep it as limited as possible.

That is exactly what I am currently implementing in trace_kprobe_klp_func_overridable().

Quoted 2 lines

Even with that, the
concerns expressed by the others still apply.

Signature · 2 lines
Regards
Yafang
#14 lore raw

RFC PATCH 3/4 livepatch: Add "replaceable" attribute to klp_patch

In reply to Yafang Shao

Add a new replaceable attribute to allow the coexistence of both atomic-replace and non-atomic-replace livepatches. If replaceable is set to 0, the livepatch will not be replaced by a subsequent atomic-replace operation.

This is a preparatory patch for following changes.

---
 include/linux/livepatch.h |  2 ++
 kernel/livepatch/core.c   | 44 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
include/linux/livepatch.h +2 -0
index ba9e3988c07c..d88a6966e5f2 100644
@@ -124,6 +124,7 @@ struct klp_state {
  * @objs:	object entries for kernel objects to be patched
  * @states:	system states that can get modified
  * @replace:	replace all actively used patches
+ * @replaceable:	whether this patch can be replaced or not
  * @list:	list node for global list of actively used patches
  * @kobj:	kobject for sysfs resources
  * @obj_list:	dynamic list of the object entries
@@ -138,6 +139,7 @@ struct klp_patch {
 	struct klp_object *objs;
 	struct klp_state *states;
 	bool replace;
+	bool replaceable;
 
 	/* internal */
 	struct list_head list;
kernel/livepatch/core.c +44 -0

67 lines not on this page · raw message

Signature · 1 line
2.47.3
maintainer
#15 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 7 lines · Yafang Shao

Add a new replaceable attribute to allow the coexistence of both atomic-replace and non-atomic-replace livepatches. If replaceable is set to 0, the livepatch will not be replaced by a subsequent atomic-replace operation.

This is a preparatory patch for following changes.

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In case we really want a feature like this, shall we add a replaceable flag to each function (klp_func)? This will give us fine granularity control. For example, user A has a non-replaceable livepatch on func_a; later user B wants to patch another version of func_a. Then some logic can detect such a conflict and reject the load of user B's livepatch. Does this make sense?

Thanks,
Song

#16 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 8 lines · Song Liu
Quoted 6 lines · Yafang Shao

Add a new replaceable attribute to allow the coexistence of both atomic-replace and non-atomic-replace livepatches. If replaceable is set to 0, the livepatch will not be replaced by a subsequent atomic-replace operation.

Quoted 6 lines
Quoted 1 line

This is a preparatory patch for following changes.

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.

However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.

Quoted 7 lines

In case we really want a feature like this, shall we add a replaceable flag to each function (klp_func)? This will give us fine granularity control. For example, user A has a non-replaceable livepatch on func_a; later user B wants to patch another version of func_a. Then some logic can detect such a conflict and reject the load of user B's livepatch. Does this make sense?

I agree with this. The ability to reject livepatches that try to change functions that have already been patched seems like an important safeguard.

Thanks,
Dylan

maintainer
#17 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 4 lines

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.

Quoted 5 lines

However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.

IIUC, the plan is:

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade "off-band' livepatches, we will have to unload the old version and load the new version later.
2. Any conflict with the off-band livepatches and regular livepatches will be difficult to manage. IOW, we kind removed the benefit of cumulative livepatches. For example, what shall we do if we really need two fixes to the same kernel functions: one from the original branch, the other from the off-band branch?

Thanks,
Song

#18 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 27 lines · Song Liu

On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 4 lines

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.

Quoted 5 lines

However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.

IIUC, the plan is:

Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

Quoted 5 lines

I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade "off-band' livepatches, we will have to unload the old version and load the new version later.

Right. That is how the non-atomic-replace patch works.

Quoted 2 lines

2. Any conflict with the off-band livepatches and regular livepatches will be difficult to manage.

We need to manage this conflict with a complex user script. That said, everything can be controlled from userspace.

Quoted 4 lines

IOW, we kind removed the benefit of cumulative
livepatches. For example, what shall we do if we really need two fixes to the same kernel functions: one from the original branch, the other from the off-band branch?

We run tens of livepatches on our production servers and have never run into this issue. It's an extremely rare case — and if it does happen, a user script should be able to handle it just fine.

Signature · 2 lines
Regards
Yafang
maintainer
#19 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 34 lines · Yafang Shao
Quoted 32 lines · Song Liu

On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 4 lines

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.

Quoted 5 lines

However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.

IIUC, the plan is:

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

Quoted 5 lines
Quoted 3 lines

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

Does this make sense?

Thanks,
Song

maintainer
#20 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 53 lines · Song Liu
Quoted 34 lines · Yafang Shao
Quoted 32 lines · Song Liu

On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 4 lines

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.

Quoted 5 lines

However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.

IIUC, the plan is:

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

Quoted 5 lines
Quoted 3 lines

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

Quoted 6 lines

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

Does this make sense?

I haven't been following the thread carefully, but could the Livepatch system state API (see Documentation/livepatch/system-state.rst) be leveraged somehow instead of adding further replace semantics?

--
Joe

maintainer
#21 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 27 lines
Quoted 27 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

Quoted 6 lines
Quoted 2 lines

Does this make sense?

I haven't been following the thread carefully, but could the Livepatch system state API (see Documentation/livepatch/system-state.rst) be leveraged somehow instead of adding further replace semantics?

AFAICT, system state will not help Yafang's use case.

Thanks,
Song

#22 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 29 lines · Song Liu

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

That sounds like a viable solution. I'll look into it and see how we can implement it.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

Does this make sense?

I haven't been following the thread carefully, but could the Livepatch system state API (see Documentation/livepatch/system-state.rst) be leveraged somehow instead of adding further replace semantics?

AFAICT, system state will not help Yafang's use case.

Right.

Signature · 2 lines
Regards
Yafang
#23 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 28 lines · Yafang Shao
Quoted 26 lines · Song Liu

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

That sounds like a viable solution. I'll look into it and see how we can implement it.

Does the following change look good to you ?

Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags

Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:

replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.

To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.

---
 .../livepatch/cumulative-patches.rst          | 20 +++++++-----
 Documentation/livepatch/livepatch.rst         | 31 +++++++++++++------
 include/linux/livepatch.h                     |  8 +++--
 kernel/livepatch/core.c                       |  4 +++
 scripts/livepatch/init.c                      |  6 +---
 scripts/livepatch/klp-build                   | 11 +++++--
 6 files changed, 53 insertions(+), 27 deletions(-)
(quoted diff) +0 -0

b/Documentation/livepatch/cumulative-patches.rst
index 1931f318976a..06e90dc5967c 100644

Documentation/livepatch/cumulative-patches.rst +1 -2
@@ -12,23 +12,26 @@ modified the same function in different ways.
 
 An elegant solution comes with the feature called "Atomic Replace". It allows
 creation of so called "Cumulative Patches". They include all wanted changes
-from all older livepatches and completely replace them in one transition.
+from older livepatches with a matching tag and replace them in one transition.
 
 Usage
 -----
 
-The atomic replace can be enabled by setting "replace" flag in struct

klp_patch,
-for example::
+he atomic replace can be enabled by setting a non-zero value to the "replace" +attribute in ``struct klp_patch``. This value acts as a **replace tag**, +defining the scope of the replacement.
+
+For example::

static struct klp_patch patch = {
.mod = THIS_MODULE,
.objs = objs,
- .replace = true,
+ .replace = 1,

        };

All processes are then migrated to use the code only from the new patch.
-Once the transition is finished, all older patches are automatically -disabled.
+Once the transition is finished, all older patches with the same replace tag +are automatically disabled. Patches with different tags remain active.

Ftrace handlers are transparently removed from functions that are no longer modified by the new cumulative patch.
@@ -62,9 +65,10 @@ Limitations:
------------

- Once the operation finishes, there is no straightforward way

-    to reverse it and restore the replaced patches atomically.
+    to reverse it and restore the replaced patches (with the same tag)
+    atomically.

- A good practice is to set .replace flag in any released livepatch.
+ A good practice is to set a consistent .replace tag in related livepatches.
Then re-adding an older livepatch is equivalent to downgrading to that patch. This is safe as long as the livepatches do _not_ do extra modifications in (un)patching callbacks or in the module_init()

(quoted diff) +0 -0

b/Documentation/livepatch/livepatch.rst
index acb90164929e..1fc1543a22c3 100644

Documentation/livepatch/livepatch.rst +22 -9
@@ -347,15 +347,28 @@ to '0'.
 5.3. Replacing
 --------------
 
-All enabled patches might get replaced by a cumulative patch that
-has the .replace flag set.
-
-Once the new patch is enabled and the 'transition' finishes then
-all the functions (struct klp_func) associated with the replaced
-patches are removed from the corresponding struct klp_ops. Also
-the ftrace handler is unregistered and the struct klp_ops is
-freed when the related function is not modified by the new patch
-and func_stack list becomes empty.
+All currently enabled patches may be superseded by a cumulative patch that
+has the ``.replace`` attribute enabled. The behavior of the replacement
+depends on the value assigned to the replace tag:
+
+replace = 0
+    Atomic replace is disabled. However, this patch remains eligible to be
+    superseded by others.
+
+replace > 0
+    Enables tagged atomic replace. Once the new patch is enabled and the
+    transition finishes, the livepatching core identifies all existing
+    patches that share the same replace tag.
+
+Once the transition is complete, all functions (``struct klp_func``)
+associated with the matching replaced patches are removed from the
+corresponding ``struct klp_ops``. If a function is no longer modified by
+the new patch and its ``func_stack`` list becomes empty, the ftrace
+handler is unregistered and the ``struct klp_ops`` is freed.
+
+Patches with a different replace tag are not affected by this process
+and remain active. This allows for the independent management and
+stacking of multiple, non-conflicting livepatch sets.
 
 See Documentation/livepatch/cumulative-patches.rst for more details.
 
include/linux/livepatch.h +6 -2
index ba9e3988c07c..417c67a17b99 100644
@@ -123,7 +123,11 @@ struct klp_state {
  * @mod:       reference to the live patch module
  * @objs:      object entries for kernel objects to be patched
  * @states:    system states that can get modified
- * @replace:   replace all actively used patches
+ * @replace:   replace tag:
+ *             = 0: Atomic replace is disabled; however, this patch remains
+ *                  eligible to be superseded by others.
+ *             > 0: Atomic replace is enabled. Only existing patches with a
+ *                  matching replace tag will be superseded.
  * @list:      list node for global list of actively used patches
  * @kobj:      kobject for sysfs resources
  * @obj_list:  dynamic list of the object entries
@@ -137,7 +141,7 @@ struct klp_patch {
        struct module *mod;
        struct klp_object *objs;
        struct klp_state *states;
-       bool replace;
+       unsigned int replace;
 
        /* internal */
        struct list_head list;
kernel/livepatch/core.c +0 -0
index 28d15ba58a26..e4e5c03b0724 100644
@@ -793,6 +793,8 @@ void klp_free_replaced_patches_async(struct

klp_patch *new_patch)

        klp_for_each_patch_safe(old_patch, tmp_patch) {
                if (old_patch == new_patch)
                        return;
+               if (old_patch->replace && old_patch->replace !=
new_patch->replace)
+                       continue;
                klp_free_patch_async(old_patch);
        }
 }
@@ -1194,6 +1196,8 @@ void klp_unpatch_replaced_patches(struct
klp_patch *new_patch)
        klp_for_each_patch(old_patch) {
                if (old_patch == new_patch)
                        return;

+ if (old_patch->replace && old_patch->replace !=
new_patch->replace)
+ continue;

                old_patch->enabled = false;
                klp_unpatch_objects(old_patch);
scripts/livepatch/init.c +1 -5
index f14d8c8fb35f..cd00e278a1d2 100644
@@ -72,11 +72,7 @@ static int __init livepatch_mod_init(void)
 
        /* TODO patch->states */
 
-#ifdef KLP_NO_REPLACE
-       patch->replace = false;
-#else
-       patch->replace = true;
-#endif
+       patch->replace = KLP_REPLACE_TAG;
 
        return klp_enable_patch(patch);
 
scripts/livepatch/klp-build +0 -0
index 7b82c7503c2b..9f6a7673304f 100755
@@ -118,6 +118,7 @@ Options:
    -j, --jobs=<jobs>           Build jobs to run simultaneously

[default: $JOBS]

    -o, --output=<file.ko>      Output file [default: livepatch-<patch-name>.ko]
        --no-replace            Disable livepatch atomic replace
+   -t, --replace-tag=<tag>     Set the atomic replace tag for this livepatch
    -v, --verbose               Pass V=1 to kernel/module builds

Advanced Options:
@@ -142,8 +143,8 @@ process_args() {
local long
local args

-       short="hfj:o:vdS:T"
-       long="help,show-first-changed,jobs:,output:,no-replace,verbose,debug,short-circuit:,keep-tmp"
+       short="hfj:o:t:vdS:T"
+       long="help,show-first-changed,jobs:,output:,no-replace,replace-tag:,verbose,debug,short-circuit:,keep-tmp"

args=$(getopt --options "$short" --longoptions "$long" -- "$@") || {
echo; usage; exit

@@ -176,6 +177,10 @@ process_args() {
                                REPLACE=0
                                shift
                                ;;
+                       -t | --replace-tag)
+                               REPLACE="$2"
+                               shift 2
+                               ;;
                        -v | --verbose)
                                VERBOSE="V=1"
                                shift

@@ -759,7 +764,7 @@ build_patch_module() {

        cflags=("-ffunction-sections")
        cflags+=("-fdata-sections")

- [[ $REPLACE -eq 0 ]] && cflags+=("-DKLP_NO_REPLACE")
+ cflags+=("-DKLP_REPLACE_TAG=$REPLACE")

        cmd=("make")
        cmd+=("$VERBOSE")

--
Regards
Yafang

maintainer
#24 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 24 lines · Yafang Shao
Quoted 23 lines · Yafang Shao
Quoted 21 lines · Song Liu

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

I still think how to make the hybrid mode more secure:

+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?

Quoted 25 lines
Quoted 2 lines

That sounds like a viable solution. I'll look into it and see how we can implement it.

Does the following change look good to you ?

Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags

Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:

replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.

To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.

include/linux/livepatch.h +0 -0
index ba9e3988c07c..417c67a17b99 100644

include/linux/livepatch.h struct klp_state { · line 128

Quoted 7 lines
include/linux/livepatch.h +3 -1
@@ -123,7 +123,11 @@ struct klp_state {
  * @mod:       reference to the live patch module
  * @objs:      object entries for kernel objects to be patched
  * @states:    system states that can get modified
- * @replace:   replace all actively used patches
+ * @replace:   replace tag:
+ *             = 0: Atomic replace is disabled; however, this patch remains
+ *                  eligible to be superseded by others.

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Do we still need this category?

include/linux/livepatch.h struct klp_patch { · line 144

Quoted 11 lines
include/linux/livepatch.h +3 -1
+ *             > 0: Atomic replace is enabled. Only existing patches with a
+ *                  matching replace tag will be superseded.
  * @list:      list node for global list of actively used patches
  * @kobj:      kobject for sysfs resources
  * @obj_list:  dynamic list of the object entries
@@ -137,7 +141,7 @@ struct klp_patch {
        struct module *mod;
        struct klp_object *objs;
        struct klp_state *states;
-       bool replace;
+       unsigned int replace;

This already breaks the backward compatibility by changing the type and semantic of this field. I would also change the name to better match the new semantic. What about using:

* @replace_set: Livepatch using the same @replace_set will get

		atomically replaced, see also conflicts[*].

	unsigned int replace_set;

[*] A livepatch module, livepatching an already livepatches function,
can be loaded only when it has the same @replace_set number.

By other words, two livepatches conflict when they have a different @replace_set number and have at least one livepatched version in common.

Quoted 27 lines
include/linux/livepatch.h +0 -0
        /* internal */
        struct list_head list;
kernel/livepatch/core.c +2 -0
index 28d15ba58a26..e4e5c03b0724 100644
@@ -793,6 +793,8 @@ void klp_free_replaced_patches_async(struct
 klp_patch *new_patch)
        klp_for_each_patch_safe(old_patch, tmp_patch) {
                if (old_patch == new_patch)
                        return;
+               if (old_patch->replace && old_patch->replace !=
 new_patch->replace)
+                       continue;
                klp_free_patch_async(old_patch);
        }

}

kernel/livepatch/core.c +2 -0
@@ -1194,6 +1196,8 @@ void klp_unpatch_replaced_patches(struct
 klp_patch *new_patch)
        klp_for_each_patch(old_patch) {
                if (old_patch == new_patch)
                        return;
+               if (old_patch->replace && old_patch->replace !=
 new_patch->replace)
+                       continue;
 
Quoted 2 lines
                old_patch->enabled = false;
                klp_unpatch_objects(old_patch);

This handles only the freeing part. More changes will be necessary:

+ klp_is_patch_compatible() must check also conflicts between livepatches with different @replace_set. The conflicts might be in the lists of:

+ livepatched functions
+ state IDs (aka callbacks and shadow variables IDs)

+ klp_add_nops() must skip livepatches with another @replace_set

+ klp_unpatch_replaced_patches() should unpatch only patches with the same @replace_set

Finally, we would need to update existing selftests plus add new selftests.

It is possible that I have missed something.

Anyway, you should wait for more feedback before you do too much coding, especially the selftests are not needed at RFC stage.

Best Regards,
Petr

maintainer
#25 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

On Tue, Apr 7, 2026 at 8:08 AM Petr Mladek <[email protected]> wrote:
[...]

Quoted 9 lines
Quoted 3 lines

+ * @replace: replace tag:
+ * = 0: Atomic replace is disabled; however, this patch remains
+ * eligible to be superseded by others.

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Do we still need this category?

Quoted 14 lines

include/linux/livepatch.h struct klp_patch { · line 144

Quoted 11 lines

+ * > 0: Atomic replace is enabled. Only existing patches with a
+ * matching replace tag will be superseded.

  * @list:      list node for global list of actively used patches
  * @kobj:      kobject for sysfs resources
  * @obj_list:  dynamic list of the object entries
include/linux/livepatch.h +1 -1
@@ -137,7 +141,7 @@ struct klp_patch {
        struct module *mod;
        struct klp_object *objs;
        struct klp_state *states;
-       bool replace;
+       unsigned int replace;

This already breaks the backward compatibility by changing the type and semantic of this field.

I was thinking if replace=0 means no replace, it is still backward compatible. Did I miss something?

Thanks,
Song

Quoted 15 lines

I would also change the name to better
match the new semantic. What about using:

* @replace_set: Livepatch using the same @replace_set will get atomically replaced, see also conflicts[*].

unsigned int replace_set;

[*] A livepatch module, livepatching an already livepatches function,
can be loaded only when it has the same @replace_set number.

By other words, two livepatches conflict when they have a different @replace_set number and have at least one livepatched version in common.

[...]

maintainer
#26 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 26 lines · Song Liu

On Tue, Apr 7, 2026 at 8:08 AM Petr Mladek <[email protected]> wrote:
[...]

Quoted 9 lines
Quoted 3 lines

+ * @replace: replace tag:
+ * = 0: Atomic replace is disabled; however, this patch remains
+ * eligible to be superseded by others.

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Do we still need this category?

Quoted 14 lines

include/linux/livepatch.h struct klp_patch { · line 144

Quoted 11 lines

+ * > 0: Atomic replace is enabled. Only existing patches with a
+ * matching replace tag will be superseded.

  * @list:      list node for global list of actively used patches
  * @kobj:      kobject for sysfs resources
  * @obj_list:  dynamic list of the object entries
include/linux/livepatch.h +1 -1
@@ -137,7 +141,7 @@ struct klp_patch {
        struct module *mod;
        struct klp_object *objs;
        struct klp_state *states;
-       bool replace;
+       unsigned int replace;

This already breaks the backward compatibility by changing the type and semantic of this field.

Quoted 2 lines

I was thinking if replace=0 means no replace, it is still backward compatible. Did I miss something?

IMHO, the semantic of the no-replace mode would be strange if we introduce the hybrid mode. Especially, it would be strange when it can be replaced by any livepatch with random replace tag/set. Also it would just complicate the definition and detection of conflicts.

I am going to provide more details in the reply to Yafang.

Best Regards,
Petr

#27 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

In reply to Petr Mladek
Quoted 31 lines · Petr Mladek
Quoted 29 lines · Yafang Shao
Quoted 28 lines · Yafang Shao
Quoted 26 lines · Song Liu

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

I still think how to make the hybrid mode more secure:

+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?

We might consider extending the klp_shadow_* API to support the new livepatch tag.

Quoted 37 lines
Quoted 26 lines
Quoted 3 lines

That sounds like a viable solution. I'll look into it and see how we can implement it.

Does the following change look good to you ?

Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags

Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:

replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.

To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.

include/linux/livepatch.h +0 -0
index ba9e3988c07c..417c67a17b99 100644

include/linux/livepatch.h struct klp_state { · line 128

Quoted 7 lines
include/linux/livepatch.h +3 -1
@@ -123,7 +123,11 @@ struct klp_state {
  * @mod:       reference to the live patch module
  * @objs:      object entries for kernel objects to be patched
  * @states:    system states that can get modified
- * @replace:   replace all actively used patches
+ * @replace:   replace tag:
+ *             = 0: Atomic replace is disabled; however, this patch remains
+ *                  eligible to be superseded by others.

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Quoted 1 line

Do we still need this category?

It can be superseded by any livepatch that has a non-zero tag set.

This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.

Quoted 14 lines

include/linux/livepatch.h struct klp_patch { · line 144

Quoted 12 lines
include/linux/livepatch.h +3 -1
+ *             > 0: Atomic replace is enabled. Only existing patches with a
+ *                  matching replace tag will be superseded.
  * @list:      list node for global list of actively used patches
  * @kobj:      kobject for sysfs resources
  * @obj_list:  dynamic list of the object entries
@@ -137,7 +141,7 @@ struct klp_patch {
        struct module *mod;
        struct klp_object *objs;
        struct klp_state *states;
-       bool replace;
+       unsigned int replace;

This already breaks the backward compatibility

It doesn't break backward compatibility.

Quoted 8 lines

by changing the type
and semantic of this field. I would also change the name to better match the new semantic. What about using:

* @replace_set: Livepatch using the same @replace_set will get atomically replaced, see also conflicts[*].

Quoted 8 lines

unsigned int replace_set;

[*] A livepatch module, livepatching an already livepatches function,
can be loaded only when it has the same @replace_set number.

By other words, two livepatches conflict when they have a different @replace_set number and have at least one livepatched version in common.

Renaming it to @replace_set makes sense to me.

Quoted 38 lines
Quoted 28 lines
include/linux/livepatch.h +0 -0
        /* internal */
        struct list_head list;
kernel/livepatch/core.c +2 -0
index 28d15ba58a26..e4e5c03b0724 100644
@@ -793,6 +793,8 @@ void klp_free_replaced_patches_async(struct
 klp_patch *new_patch)
        klp_for_each_patch_safe(old_patch, tmp_patch) {
                if (old_patch == new_patch)
                        return;
+               if (old_patch->replace && old_patch->replace !=
 new_patch->replace)
+                       continue;
                klp_free_patch_async(old_patch);
        }

}

kernel/livepatch/core.c +2 -0
@@ -1194,6 +1196,8 @@ void klp_unpatch_replaced_patches(struct
 klp_patch *new_patch)
        klp_for_each_patch(old_patch) {
                if (old_patch == new_patch)
                        return;
+               if (old_patch->replace && old_patch->replace !=
 new_patch->replace)
+                       continue;
 
Quoted 2 lines
                old_patch->enabled = false;
                klp_unpatch_objects(old_patch);

This handles only the freeing part. More changes will be necessary:

+ klp_is_patch_compatible() must check also conflicts between livepatches with different @replace_set. The conflicts might be in the lists of:

Quoted 7 lines

+ livepatched functions
+ state IDs (aka callbacks and shadow variables IDs)

+ klp_add_nops() must skip livepatches with another @replace_set

+ klp_unpatch_replaced_patches() should unpatch only patches with the same @replace_set

I appreciate the guidance on this.

Quoted 8 lines

Finally, we would need to update existing selftests plus add new selftests.

It is possible that I have missed something.

Anyway, you should wait for more feedback before you do too much coding, especially the selftests are not needed at RFC stage.

Sure.

Signature · 2 lines
Regards
Yafang
maintainer
#28 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 36 lines · Yafang Shao
Quoted 35 lines · Petr Mladek
Quoted 32 lines · Yafang Shao
Quoted 31 lines · Yafang Shao
Quoted 29 lines · Song Liu

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

Quoted 7 lines
Quoted 4 lines

I still think how to make the hybrid mode more secure:

+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?

We might consider extending the klp_shadow_* API to support the new livepatch tag.

It would be nice to associate shadow variables with states so that we could check which shadow variables are used by each livepatch.

It is partially implemented in my earlier RFC, see
https://lore.kernel.org/all/[email protected]/

Quoted 36 lines
Quoted 36 lines
Quoted 25 lines
Quoted 2 lines

That sounds like a viable solution. I'll look into it and see how we can implement it.

Does the following change look good to you ?

Subject: [PATCH] livepatch: Support scoped atomic replace using replace tags

Extend the replace attribute from a boolean to a u32 to act as a replace tag. This introduces the following semantics:

replace = 0: Atomic replace is disabled. However, this patch remains
eligible to be superseded by others.
replace > 0: Enables tagged replace (default is 1). A newly loaded
livepatch will only replace existing patches that share the same tag.

To maintain backward compatibility, a patch with replace == 0 does not trigger an outgoing atomic replace, but remains eligible to be superseded by any incoming patch with a valid replace tag.

include/linux/livepatch.h +0 -0
index ba9e3988c07c..417c67a17b99 100644

include/linux/livepatch.h struct klp_state { · line 128

Quoted 7 lines
include/linux/livepatch.h +3 -1
@@ -123,7 +123,11 @@ struct klp_state {
  * @mod:       reference to the live patch module
  * @objs:      object entries for kernel objects to be patched
  * @states:    system states that can get modified
- * @replace:   replace all actively used patches
+ * @replace:   replace tag:
+ *             = 0: Atomic replace is disabled; however, this patch remains
+ *                  eligible to be superseded by others.

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Quoted 3 lines
Quoted 1 line

Do we still need this category?

It can be superseded by any livepatch that has a non-zero tag set.

And this exactly the weird thing.

A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.

Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.

Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.

So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.

Quoted 4 lines

This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.

IMHO, the backward compatibility would just create complexity and mess in this case.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
include/linux/livepatch.h +2 -0
+ *             > 0: Atomic replace is enabled. Only existing patches with a
+ *                  matching replace tag will be superseded.
  * @list:      list node for global list of actively used patches
  * @kobj:      kobject for sysfs resources
  * @obj_list:  dynamic list of the object entries
Quoted 8 lines

include/linux/livepatch.h struct klp_patch { · line 144

Quoted 6 lines
include/linux/livepatch.h +1 -1
@@ -137,7 +141,7 @@ struct klp_patch {
        struct module *mod;
        struct klp_object *objs;
        struct klp_state *states;
-       bool replace;
+       unsigned int replace;

This already breaks the backward compatibility

Quoted 1 line

It doesn't break backward compatibility.

It does. Livepatches with .replace flag set would need to define:

	struct livepatch patch = {
		.replace = <number>,
	}

instead of

	struct livepatch patch = {
		.replace = true,
	}

Best Regards,
Petr

maintainer
#29 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

On Wed, Apr 8, 2026 at 4:43 AM Petr Mladek <[email protected]> wrote:
[...]

Quoted 19 lines
Quoted 7 lines
Quoted 5 lines

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Do we still need this category?

It can be superseded by any livepatch that has a non-zero tag set.

And this exactly the weird thing.

A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.

Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.

Quoted 8 lines

Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.

So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.

I think I see your point now. Existing code works as:
- replace=false doesn't replace anything
- replace=true replaces everything

If we assume false=0 and true=1, it is technically possible to define:
- replace_set=0 doesn't replace anything
- replace_set=1 replaces everything
- replace_set=2+ only replace the same replace_set

This is probably a little too complicated.

Quoted 7 lines
Quoted 4 lines

This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.

IMHO, the backward compatibility would just create complexity and mess in this case.

Given that livepatch is for expert users, I think we can make this work without backward compatibility. But breaking compatibility is always not preferred.

Thanks,
Song

maintainer
#30 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 30 lines · Song Liu

On Wed, Apr 8, 2026 at 4:43 AM Petr Mladek <[email protected]> wrote:
[...]

Quoted 19 lines
Quoted 7 lines
Quoted 5 lines

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Do we still need this category?

It can be superseded by any livepatch that has a non-zero tag set.

And this exactly the weird thing.

A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.

Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.

Quoted 8 lines

Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.

So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.

Quoted 8 lines

I think I see your point now. Existing code works as:
- replace=false doesn't replace anything
- replace=true replaces everything

If we assume false=0 and true=1, it is technically possible to define:
- replace_set=0 doesn't replace anything
- replace_set=1 replaces everything
- replace_set=2+ only replace the same replace_set

Yes. This well describes my point.

Quoted 7 lines

This is probably a little too complicated.

Quoted 5 lines
Quoted 4 lines

This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.

Quoted 6 lines
Quoted 2 lines

IMHO, the backward compatibility would just create complexity and mess in this case.

Given that livepatch is for expert users, I think we can make this work without backward compatibility. But breaking compatibility is always not preferred.

I believe that it is acceptable because:

1. It was always hard to combine no-replace and replace livepatches. I wonder if anyone combines them at all.

2. I believe that nobody tries to load the same livepatch module on different kernel versions. Instead, everyone prepares a custom livepatch module for each livepatched kernel version/release.

And the tooling for creating livepatches will need to be updated to use "number" instead of "true/false" anyway.

That said, it is easier to always use "0" for non-replace patches instead of assigning an unique "number" to avoid replacing. But I do not think that this would justify the complexity of having different semantic for 0, 1, and 2+ replace_set numbers.

Best Regards,
Petr

#31 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

Quoted 53 lines · Petr Mladek
Quoted 30 lines · Song Liu

On Wed, Apr 8, 2026 at 4:43 AM Petr Mladek <[email protected]> wrote:
[...]

Quoted 19 lines
Quoted 7 lines
Quoted 5 lines

This is weird semantic. Which livepatch tag would be allowed to supersede it, please?

Do we still need this category?

It can be superseded by any livepatch that has a non-zero tag set.

And this exactly the weird thing.

A patch with the .replace flag set is supposed to obsolete all already installed livepatches. It means that it should provide all existing fixes and features.

Now, we want to introduce a replace flag/set which would allow to replace/obsolete only the livepatch with the same tag/set number. And we want to prevent conflicts by making sure that livepatches with different tag/set number will never livepatch the same function.

Quoted 8 lines

Obviously, livepatches with different tag/set number could not obsolete the same no-replace livepatch. They would need to livepatch the same functions touched by the no-replace livepatch and would conflict.

So, I suggest to remove the no-replace mode completely. It should not be needed. A livepatch which should be installed in parallel will simply use another unique tag/set number.

Quoted 8 lines

I think I see your point now. Existing code works as:
- replace=false doesn't replace anything
- replace=true replaces everything

If we assume false=0 and true=1, it is technically possible to define:
- replace_set=0 doesn't replace anything
- replace_set=1 replaces everything
- replace_set=2+ only replace the same replace_set

Yes. This well describes my point.

Quoted 10 lines

This is probably a little too complicated.

Quoted 7 lines
Quoted 4 lines

This ensures backward compatibility: while a non-atomic-replace livepatch can be superseded by an atomic-replace one, the reverse is not permitted—an atomic-replace livepatch cannot be superseded by a non-atomic one.

IMHO, the backward compatibility would just create complexity and mess in this case.

Quoted 8 lines
Quoted 3 lines

Given that livepatch is for expert users, I think we can make this work without backward compatibility. But breaking compatibility is always not preferred.

I believe that it is acceptable because:

1. It was always hard to combine no-replace and replace livepatches. I wonder if anyone combines them at all.

Because 'replace' patches can supersede 'no-replace' ones, users have to maintain a strict loading order. I doubt anyone actually combines them in production.

Quoted 4 lines

2. I believe that nobody tries to load the same livepatch module on different kernel versions. Instead, everyone prepares a custom livepatch module for each livepatched kernel version/release.

Correct. We always build and apply distinct livepatches for each specific kernel version.

Quoted 8 lines

And the tooling for creating livepatches will need to be updated to use "number" instead of "true/false" anyway.

That said, it is easier to always use "0" for non-replace patches instead of assigning an unique "number" to avoid replacing. But I do not think that this would justify the complexity of having different semantic for 0, 1, and 2+ replace_set numbers.

Fair enough. Let's drop backward compatibility to keep the implementation simple.

--
Regards
Yafang

#32 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

In reply to Petr Mladek
Quoted 43 lines · Petr Mladek
Quoted 41 lines · Yafang Shao
Quoted 39 lines · Petr Mladek
Quoted 24 lines · Yafang Shao
Quoted 23 lines · Yafang Shao
Quoted 21 lines · Song Liu

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <[email protected]> wrote:
[...]

Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce some replace tag concept: Each livepatch will have a tag, u32 number. Newly loaded livepatch will only replace existing livepatch with the same tag. We can even reuse the existing "bool replace" in klp_patch, and make it u32: replace=0 means no replace; replace > 0 are the replace tag.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

For current users of cumulative patches, all the livepatch will have the same tag, say 1. For your use case, you can assign each user a unique tag. Then all these users can do atomic upgrades of their own livepatches.

We may also need to check whether two livepatches of different tags touch the same kernel function. When that happens, the later livepatch should fail to load.

I still think how to make the hybrid mode more secure:

+ The isolated sets of livepatched functions look like a good rule.
+ What about isolating the shadow variables/states as well?

Quoted 8 lines
Quoted 2 lines

We might consider extending the klp_shadow_* API to support the new livepatch tag.

It would be nice to associate shadow variables with states so that we could check which shadow variables are used by each livepatch.

It is partially implemented in my earlier RFC, see
https://lore.kernel.org/all/[email protected]/

This patch is still pending acceptance, but it offers a nice improvement. With your modifications, the remaining task would be to integrate a new replace_set into klp_state and update the API accordingly

[...]

Signature · 2 lines
Regards
Yafang
maintainer
#33 lore raw

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

In reply to Yafang Shao
Quoted 53 lines · Yafang Shao
Quoted 27 lines · Song Liu

On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 4 lines

IIRC, the use case for this change is when multiple users load various livepatch modules on the same system. I still don't believe this is the right way to manage livepatches. That said, I won't really NACK this if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch module, and we use per-kernel build "livepatch release" branches to track the contents of these cumulative livepatches. This model has worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to certain subpopulations of production. If the subpopulation is running the same build of everything else, this would require us to introduce another branching factor to the "livepatch release" branches -- something we do not support due to the added toil and complexity.

Quoted 5 lines

However, if we had the ability to build "off-band" livepatch modules that were marked as non-replaceable, we could support these selective patches without the additional branching factor. I will have to circulate the idea internally, but to me this seems like a very useful option to have in certain cases.

IIUC, the plan is:

Quoted 8 lines

- The regular livepatches are cumulative, have the replace flag; and are replaceable.
- The occasional "off-band" livepatches do not have the replace flag, and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can still release a cumulative livepatch to replace the previous cumulative livepatch. Is this the expected use case?

That matches our expected use case.

Quoted 5 lines

I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade "off-band' livepatches, we will have to unload the old version and load the new version later.

Right. That is how the non-atomic-replace patch works.

Quoted 2 lines

2. Any conflict with the off-band livepatches and regular livepatches will be difficult to manage.

We need to manage this conflict with a complex user script. That said, everything can be controlled from userspace.

Quoted 8 lines
Quoted 4 lines

IOW, we kind removed the benefit of cumulative
livepatches. For example, what shall we do if we really need two fixes to the same kernel functions: one from the original branch, the other from the off-band branch?

We run tens of livepatches on our production servers and have never run into this issue. It's an extremely rare case — and if it does happen, a user script should be able to handle it just fine.

Could you please share the script? Or at least summarize the situations when this script detect a conflict and refuse loading a livepatch?

I believe that most/all of these checks can be implemented in the kernel. And if we agreed to add a hybrid mode than it should be added together with the checks.

We have already invested a lot of effort into make the kernel livepatching as safe as possible. From my POV, the most important parts are:

+ consistency model: Tasks are transitioned separately when they do not use any livepatched function.

+ atomic replace: Transition all livepatched functions at once.

If we agree to add the hybrid model then we should add it with some safety belts as well. And it would be nice to get inspiration about the safety checks from your script.

Best Regards,
Petr

#34 lore raw

RFC PATCH 4/4 livepatch: Implement livepatch hybrid mode

In reply to Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.

To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.

---
 kernel/livepatch/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/livepatch/core.c +6 -0
index 04f9e84f114f..5a44154131c8 100644
@@ -665,6 +665,8 @@ static int klp_add_nops(struct klp_patch *patch)
 		klp_for_each_object(old_patch, old_obj) {
 			int err;
 
+			if (!old_patch->replaceable)
+				continue;
 			err = klp_add_object_nops(patch, old_obj);
 			if (err)
 				return err;
@@ -837,6 +839,8 @@ void klp_free_replaced_patches_async(struct klp_patch *new_patch)
 	klp_for_each_patch_safe(old_patch, tmp_patch) {
 		if (old_patch == new_patch)
 			return;
+		if (!old_patch->replaceable)
+			continue;
 		klp_free_patch_async(old_patch);
 	}
 }
@@ -1239,6 +1243,8 @@ void klp_unpatch_replaced_patches(struct klp_patch *new_patch)
 		if (old_patch == new_patch)
 			return;
 
+		if (!old_patch->replaceable)
+			continue;
 		old_patch->enabled = false;
 		klp_unpatch_objects(old_patch);
 	}
Signature · 1 line
2.47.3
maintainer
#35 lore raw
In reply to Yafang Shao

Hi Yafang,

Quoted 8 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

Quoted 7 lines

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.

Thanks,
Song

#36 lore raw
Quoted 17 lines · Song Liu

Hi Yafang,

Quoted 13 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

Quoted 9 lines
Quoted 2 lines

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

Signature · 2 lines
Regards
Yafang
maintainer
#37 lore raw
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu

Hi Yafang,

Quoted 13 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

Quoted 9 lines
Quoted 2 lines

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

Quoted 7 lines

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.

Can we use struct_ops instead of bpf_override_return for this case? This should make the solution more flexible.

Thanks,
Song

#38 lore raw
Quoted 43 lines · Song Liu
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu

Hi Yafang,

Quoted 13 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

Quoted 9 lines
Quoted 2 lines

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

Quoted 7 lines

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

Quoted 3 lines

I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.

Right

Quoted 3 lines

Can we use struct_ops instead of bpf_override_return for this case? This should make the solution more flexible.

Upstreaming struct_ops based BPF hooks is a challenging process, as seen in these examples:

https://lwn.net/Articles/1054030/
https://lwn.net/Articles/1043548/

Even when successful, upstreaming can take a significant amount of time—often longer than our production requirements allow. To bridge this gap, we developed this livepatch+BPF solution. This allows us to rapidly deploy new features without maintaining custom hooks in our local kernel. Because these livepatch-based hooks are lightweight, they minimize maintenance overhead and simplify kernel upgrades (e.g., from 6.1 to 6.18).

That said, we would still prefer to have our hooks accepted upstream to eliminate the need for self-maintenance entirely.

Signature · 2 lines
Regards
Yafang
maintainer
#39 lore raw
Quoted 61 lines · Yafang Shao
Quoted 43 lines · Song Liu
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu

Hi Yafang,

Quoted 13 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

Quoted 9 lines
Quoted 2 lines

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

Quoted 7 lines

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

Quoted 3 lines

I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.

Right

Quoted 3 lines

Can we use struct_ops instead of bpf_override_return for this case? This should make the solution more flexible.

Upstreaming struct_ops based BPF hooks is a challenging process, as seen in these examples:

https://lwn.net/Articles/1054030/
https://lwn.net/Articles/1043548/

Quoted 7 lines

Even when successful, upstreaming can take a significant amount of time—often longer than our production requirements allow. To bridge this gap, we developed this livepatch+BPF solution. This allows us to rapidly deploy new features without maintaining custom hooks in our local kernel. Because these livepatch-based hooks are lightweight, they minimize maintenance overhead and simplify kernel upgrades (e.g., from 6.1 to 6.18).

I didn't mean upstream struct_ops.

We can define the struct_ops in an OOT kernel module. Then we can attach BPF programs to the struct_ops. We may need livepatch to connect the new struct_ops to original kernel logic.

I think kernel side of this solution is mostly available, but we may need some work on the toolchain side.

Does this make sense?

Thanks,
Song

Quoted 2 lines

That said, we would still prefer to have our hooks accepted upstream to eliminate the need for self-maintenance entirely.

#40 lore raw
Quoted 73 lines · Song Liu
Quoted 61 lines · Yafang Shao
Quoted 43 lines · Song Liu
Quoted 33 lines · Yafang Shao
Quoted 17 lines · Song Liu

Hi Yafang,

Quoted 13 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

Quoted 9 lines
Quoted 2 lines

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this? AFAICT, livepatch is more flexible but risky (may cause crash); while BPF is safe, but less flexible. The combination you are proposing seems to get the worse of the two sides. Maybe it can indeed get the benefit of both sides in some cases, but I cannot think of such examples.

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

Quoted 7 lines

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

Quoted 3 lines

I guess the idea here is: keep the risk part simple, and implement it in module/livepatch, then use BPF for the flexible and programmable part safe.

Right

Quoted 3 lines

Can we use struct_ops instead of bpf_override_return for this case? This should make the solution more flexible.

Upstreaming struct_ops based BPF hooks is a challenging process, as seen in these examples:

https://lwn.net/Articles/1054030/
https://lwn.net/Articles/1043548/

Quoted 7 lines

Even when successful, upstreaming can take a significant amount of time—often longer than our production requirements allow. To bridge this gap, we developed this livepatch+BPF solution. This allows us to rapidly deploy new features without maintaining custom hooks in our local kernel. Because these livepatch-based hooks are lightweight, they minimize maintenance overhead and simplify kernel upgrades (e.g., from 6.1 to 6.18).

I didn't mean upstream struct_ops.

Quoted 8 lines

We can define the struct_ops in an OOT kernel module. Then we can attach BPF programs to the struct_ops. We may need livepatch to connect the new struct_ops to original kernel logic.

I think kernel side of this solution is mostly available, but we may need some work on the toolchain side.

Does this make sense?

Are there actual benefits to using struct_ops instead of bpf_override_return? So far, I’ve only found it adds complexity without much gain.
Can we add something like ALLOW_LIVEPATCH_ERROR_INJECTION() to allow error injection on functions defined inside a livepatch?

Signature · 2 lines
Regards
Yafang
maintainer
#41 lore raw

On Mon, Apr 6, 2026 at 8:14 PM Yafang Shao <[email protected]> wrote:
[...]

Quoted 4 lines
Quoted 4 lines

We can define the struct_ops in an OOT kernel module. Then we can attach BPF programs to the struct_ops. We may need livepatch to connect the new struct_ops to original kernel logic.

Quoted 8 lines
Quoted 4 lines

I think kernel side of this solution is mostly available, but we may need some work on the toolchain side.

Does this make sense?

Are there actual benefits to using struct_ops instead of bpf_override_return? So far, I’ve only found it adds complexity without much gain.

Yes, struct_ops can be more powerful. For example, we can use something like the following to modify the content of /proc/cmdline with a bpf program. AFAICT, this is not possible with bpf_override_return.

Note that this example doesn't require kernel changes. This is actually a fun example. I will send the full code as a new self test.

Thanks,
Song

======= key logic of the livepatch module =======

static int livepatch_cmdline_proc_show(struct seq_file *m, void *v)

{
        struct klp_bpf_cmdline_ops *ops = READ_ONCE(active_ops);

        if (ops && ops->set_cmdline)
                return ops->set_cmdline(m);

        seq_printf(m, "%s: no struct_ops attached\n", THIS_MODULE->name);
        return 0;
}

static struct klp_func funcs[] = {
{
.old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show,
}, { }
};

========== key logic of the bpf program =========

SEC("struct_ops/set_cmdline")
int BPF_PROG(set_cmdline, struct seq_file *m)

{
        char custom[] = "klp_bpf: custom cmdline\n";

        bpf_klp_seq_write(m, custom, sizeof(custom) - 1);
        return 0;
}

SEC(".struct_ops.link")
struct klp_bpf_cmdline_ops cmdline_ops = {
.set_cmdline = (void *)set_cmdline,
};

maintainer
#42 lore raw
In reply to Yafang Shao
Quoted 2 lines

Can we add something like ALLOW_LIVEPATCH_ERROR_INJECTION() to allow error injection on functions defined inside a livepatch?

No.

I am sorry but you always seem to find band aids to your set up and how you deal with live patches internally. While I can see that something like a hybrid mode might be useful to people if done right (and we are not there yet), the combination of it with bpf overrides or anything like that is not something I would like to see in upstream.

Miroslav

#43 lore raw
Quoted 11 lines · Miroslav Benes
Quoted 3 lines

Can we add something like ALLOW_LIVEPATCH_ERROR_INJECTION() to allow error injection on functions defined inside a livepatch?

No.

I am sorry but you always seem to find band aids to your set up and how you deal with live patches internally. While I can see that something like a hybrid mode might be useful to people if done right (and we are not there yet), the combination of it with bpf overrides or anything like that is not something I would like to see in upstream.

The upstream kernel already allows for combining BPF and livepatch to override functions. Song’s patch offers a great reference for implementing this without changing the kernel:

https://lore.kernel.org/bpf/[email protected]/

Signature · 2 lines
Regards
Yafang
#44 lore raw
In reply to Yafang Shao
Quoted 2 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads.

Myabe it allows, or based on the rest of the mail not quite. But that is certainly not the intent at all, the intent is to fix critical bugs without downtime.

Quoted 6 lines

However, static livepatches lack
the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

Quoted 7 lines

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Whol f**. now. Is this a delayed April 1st post?

#45 lore raw
Quoted 18 lines · Christoph Hellwig
Quoted 2 lines · Yafang Shao

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads.

Myabe it allows, or based on the rest of the mail not quite. But that is certainly not the intent at all, the intent is to fix critical bugs without downtime.

Quoted 9 lines

However, static livepatches lack
the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

Quoted 6 lines
Quoted 4 lines

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Whol f**. now. Is this a delayed April 1st post?

You're already in my spam list. Don't expect any further replies. Feel free to keep your verbose rubbish to yourself.

Signature · 2 lines
Regards
Yafang
#46 lore raw
In reply to Yafang Shao

Hi Yafang,

Quoted 7 lines · On Thu, 2 Apr 2026 17:26:03 +0800 Yafang Shao <[email protected]> wrote:

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

Quoted 7 lines

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

First of all, I don't like this approach to test a new feature in the kernel, because it sounds like allowing multiple different generations of implementations to coexist simultaneously. The standard kernel code is not designed to withstand such implementations.

For example, if you implement a well-designed framework in a specific subsystem, like Schedext, which allows multiple implementations extended with BPF to coexist, there's no problem (at least it's debatable).

But if it is for any function, it is dangerous feature. Bugs that occur in kernels that use this functionality cannot be addressed here. They need to be treated the same way as out-of-tree drivers or forked kernels. I mean, add a tainted flag for this feature. And we don't care of it.

Quoted 9 lines

A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.

To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.

Can you share your actual problem to be solved?
If the specific problem and the specific subsystem are clear, I think there is room to discuss it with the subsystem maintainer.

Quoted 6 lines

Furthermore, this mechanism provides a lower-maintenance alternative to out-of-tree BPF hooks. Given the complexities of upstreaming custom BPF hooks (e.g., [0], [1]), this hybrid mode allows for the maintenance of stable, minimal hook points via livepatching with significantly reduced maintenance burden.

Maintenance cost is the same. We need to add out-of-tree BPF hooks on source code. Maybe deploying cost will be reduced.

Thank you,

Quoted 21 lines

Yafang Shao (4):
trace: Simplify kprobe overridable function check
trace: Allow kprobes to override livepatched functions
livepatch: Add "replaceable" attribute to klp_patch
livepatch: Implement livepatch hybrid mode

 include/linux/livepatch.h   |  2 ++
 kernel/livepatch/core.c     | 50 +++++++++++++++++++++++++++++++
 kernel/trace/Kconfig        | 14 +++++++++
 kernel/trace/bpf_trace.c    | 14 ++++++---
 kernel/trace/trace_kprobe.c | 49 ++++++++++++------------------
 kernel/trace/trace_probe.h  | 59 +++++++++++++++++++++++++++----------
 6 files changed, 139 insertions(+), 49 deletions(-)
Signature · 1 line
2.47.3
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#47 lore raw
Quoted 18 lines · Masami Hiramatsu

Hi Yafang,

Quoted 12 lines · On Thu, 2 Apr 2026 17:26:03 +0800 Yafang Shao <[email protected]> wrote:

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

Quoted 7 lines
Quoted 2 lines

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

First of all, I don't like this approach to test a new feature in the kernel, because it sounds like allowing multiple different generations of implementations to coexist simultaneously. The standard kernel code is not designed to withstand such implementations.

However, this approach is invaluable for rapidly deploying new kernel features to production servers without downtime. Upgrading kernels across a large fleet remains a significant challenge.

Quoted 9 lines

For example, if you implement a well-designed framework in a specific subsystem, like Schedext, which allows multiple implementations extended with BPF to coexist, there's no problem (at least it's debatable).

But if it is for any function, it is dangerous feature. Bugs that occur in kernels that use this functionality cannot be addressed here. They need to be treated the same way as out-of-tree drivers or forked kernels. I mean, add a tainted flag for this feature. And we don't care of it.

Agreed. This should be handled as an OOT module rather than part of the core kernel.

Quoted 7 lines
Quoted 7 lines

A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.

Quoted 5 lines
Quoted 3 lines

To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.

Can you share your actual problem to be solved?

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

[...]

Signature · 2 lines
Regards
Yafang
#48 lore raw
Quoted 22 lines · On Sun, 12 Apr 2026 21:50:31 +0800 Yafang Shao <[email protected]> wrote:
Quoted 21 lines · Masami Hiramatsu

Hi Yafang,

Quoted 14 lines · On Thu, 2 Apr 2026 17:26:03 +0800 Yafang Shao <[email protected]> wrote:

Livepatching allows for rapid experimentation with new kernel features without interrupting production workloads. However, static livepatches lack the flexibility required to tune features based on task-specific attributes, such as cgroup membership, which is critical in multi-tenant k8s environments. Furthermore, hardcoding logic into a livepatch prevents dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value of that hook based on the current task's context.

Quoted 8 lines
Quoted 4 lines

First of all, I don't like this approach to test a new feature in the kernel, because it sounds like allowing multiple different generations of implementations to coexist simultaneously. The standard kernel code is not designed to withstand such implementations.

However, this approach is invaluable for rapidly deploying new kernel features to production servers without downtime. Upgrading kernels across a large fleet remains a significant challenge.

I think that downtime should be accepted as a cost for stability in general. If your new kernel feature has a bug and causes a crash, anyway it gets your servers down.

Quoted 30 lines
Quoted 9 lines

For example, if you implement a well-designed framework in a specific subsystem, like Schedext, which allows multiple implementations extended with BPF to coexist, there's no problem (at least it's debatable).

But if it is for any function, it is dangerous feature. Bugs that occur in kernels that use this functionality cannot be addressed here. They need to be treated the same way as out-of-tree drivers or forked kernels. I mean, add a tainted flag for this feature. And we don't care of it.

Agreed. This should be handled as an OOT module rather than part of the core kernel.

Quoted 7 lines
Quoted 7 lines

A significant challenge arises when atomic-replace is enabled. In this mode, deploying a new livepatch changes the target function's address, forcing a re-attachment of the BPF program. This re-attachment latency is unacceptable in critical paths, such as those handling networking policies.

Quoted 5 lines
Quoted 3 lines

To solve this, we introduce a hybrid livepatch mode that allows specific patches to remain non-replaceable, ensuring the function address remains stable and the BPF program stays attached.

Can you share your actual problem to be solved?

Here is an example we recently deployed on our production servers:

https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

Quoted 7 lines

In one of our specific clusters, we needed to send BGP traffic out through specific NICs based on the destination IP. To achieve this without interrupting service, we live-patched bond_xmit_3ad_xor_slave_get(), added a new hook called bond_get_slave_hook(), and then ran a BPF program attached to that hook to select the outgoing NIC from the SKB. This allowed us to rapidly deploy the feature with zero downtime.

In this case, you can make specific livepatch or kernel module to replace the kernel function without using BPF on your server.

The BGP trafic in bonding device seems very specific, so it may not cause a trouble, but this is very generic change, which allows user to change more core kernel feature, e.g. memory management or scheduler etc.

Excessive degrees of freedom introduce uncertainty and instability into a system. While the functionality is interesting, it would be a way to generalize schedext in an uncontrolled way.

At a minimum, some form of build time and runtime constraint, along with a taint flag that clearly indicates in the crash logs that this feature is being used, would be necessary. (It means this is should not be used in production environment.)

Thank you,

Quoted 6 lines

[...]

Signature · 2 lines
Regards
Yafang
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>