Skip to contentSkip to conversation

PATCH In review [email protected] 5 participants 10 messages Jul 14, 2024 – Jul 25, 2024 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

[PATCH] livepatch: support of modifying refcount_t without underflow after unpatch

CVE fixes sometimes add refcount_inc/dec() pairs to the code with existing refcount_t.
Two problems arise when applying live-patch in this case:
1) After refcount_t is being inc() during system is live-patched, after unpatch the counter value will not be valid, as corresponing dec() would never be called.
2) Underflows are possible in runtime in case dec() is called before corresponding inc() in the live-patched code.

Proposed kprefcount_t functions are using following approach to solve these two problems:
1) In addition to original refcount_t, temporary refcount_t is allocated, and after unpatch it is just removed. This way system is safe with correct refcounting while patch is applied, and no underflow would happend after unpatch.
2) For inc/dec() added by live-patch code, one bit in reference-holder structure is used (unsigned char *ref_holder, kprefholder_flag). In case dec() is called first, it is just ignored as ref_holder bit would still not be initialized.

API is defined include/linux/livepatch_refcount.h:

typedef struct kprefcount_struct {
	refcount_t *refcount;
	refcount_t kprefcount;
	spinlock_t lock;
} kprefcount_t;

kprefcount_t *kprefcount_alloc(refcount_t *refcount, gfp_t flags);
void kprefcount_free(kprefcount_t *kp_ref);
int kprefcount_read(kprefcount_t *kp_ref);
void kprefcount_inc(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); void kprefcount_dec(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); bool kprefcount_dec_and_test(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);

#1 lore raw

PATCH 1/2 livepatch: support of modifying refcount_t without underflow after unpatch.

From: Roman Rashchupkin <[email protected]>

CVE fixes sometimes add refcount_inc/dec() pairs to the code with existing refcount_t. Two problems arise when applying live-patch in this case:
1) After refcount_t is being inc() during system is live-patched, after unpatch the counter value will not be valid, as corresponing dec() would never be called.
2) Underflows are possible in runtime in case dec() is called before corresponding inc() in the live-patched code.

Proposed kprefcount_t functions are using following approach to solve these two problems:
1) In addition to original refcount_t, temporary refcount_t is allocated, and after unpatch it is just removed. This way system is safe with correct refcounting while patch is applied, and no underflow would happend after unpatch.
2) For inc/dec() added by live-patch code, one bit in reference-holder structure is used (unsigned char *ref_holder, kprefholder_flag). In case dec() is called first, it is just ignored as ref_holder bit would still not be initialized.

---
 include/linux/livepatch_refcount.h | 19 +++++++
 kernel/livepatch/Makefile          |  2 +-
 kernel/livepatch/kprefcount.c      | 89 ++++++++++++++++++++++++++++++
 3 files changed, 109 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/livepatch_refcount.h
 create mode 100644 kernel/livepatch/kprefcount.c
include/linux/livepatch_refcount.h new file +19 -0
new file mode 100644
index 000000000000..02f9e7eeadb2
@@ -0,0 +1,19 @@
+#ifndef __KP_REFCOUNT_T__
+#define __KP_REFCOUNT_T__
+
+#include <linux/refcount.h>
+
+typedef struct kprefcount_struct {
+	refcount_t *refcount;
+	refcount_t kprefcount;
+	spinlock_t lock;
+} kprefcount_t;
+
+kprefcount_t *kprefcount_alloc(refcount_t *refcount, gfp_t flags);
+void kprefcount_free(kprefcount_t *kp_ref);
+int kprefcount_read(kprefcount_t *kp_ref);
+void kprefcount_inc(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);
+void kprefcount_dec(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);
+bool kprefcount_dec_and_test(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);
+
+#endif
kernel/livepatch/Makefile +1 -1
index cf03d4bdfc66..8ff0926372c2 100644
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_LIVEPATCH) += livepatch.o
 
-livepatch-objs := core.o patch.o shadow.o state.o transition.o
+livepatch-objs := core.o patch.o shadow.o state.o transition.o kprefcount.o
kernel/livepatch/kprefcount.c new file +89 -0

89 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/2] [PATCH] livepatch: support of modifying refcount_t without underflow after unpatch.

Quoted 68 lines · On 7/14/24 12:59, [email protected] wrote:

From: Roman Rashchupkin <[email protected]>

CVE fixes sometimes add refcount_inc/dec() pairs to the code with existing refcount_t. Two problems arise when applying live-patch in this case:
1) After refcount_t is being inc() during system is live-patched, after unpatch the counter value will not be valid, as corresponing dec() would never be called.
2) Underflows are possible in runtime in case dec() is called before corresponding inc() in the live-patched code.

Proposed kprefcount_t functions are using following approach to solve these two problems:
1) In addition to original refcount_t, temporary refcount_t is allocated, and after unpatch it is just removed. This way system is safe with correct refcounting while patch is applied, and no underflow would happend after unpatch.
2) For inc/dec() added by live-patch code, one bit in reference-holder structure is used (unsigned char *ref_holder, kprefholder_flag). In case dec() is called first, it is just ignored as ref_holder bit would still not be initialized.

---
  include/linux/livepatch_refcount.h | 19 +++++++
  kernel/livepatch/Makefile          |  2 +-
  kernel/livepatch/kprefcount.c      | 89 ++++++++++++++++++++++++++++++
  3 files changed, 109 insertions(+), 1 deletion(-)

create mode 100644 include/linux/livepatch_refcount.h create mode 100644 kernel/livepatch/kprefcount.c

include/linux/livepatch_refcount.h new file +19 -0
new file mode 100644
index 000000000000..02f9e7eeadb2
@@ -0,0 +1,19 @@
+#ifndef __KP_REFCOUNT_T__
+#define __KP_REFCOUNT_T__
+
+#include <linux/refcount.h>
+
+typedef struct kprefcount_struct {
+	refcount_t *refcount;
+	refcount_t kprefcount;
+	spinlock_t lock;
+} kprefcount_t;
+
+kprefcount_t *kprefcount_alloc(refcount_t *refcount, gfp_t flags);
+void kprefcount_free(kprefcount_t *kp_ref);
+int kprefcount_read(kprefcount_t *kp_ref);
+void kprefcount_inc(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);
+void kprefcount_dec(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);
+bool kprefcount_dec_and_test(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);
+
+#endif
kernel/livepatch/Makefile +1 -1
index cf03d4bdfc66..8ff0926372c2 100644
@@ -1,4 +1,4 @@
  # SPDX-License-Identifier: GPL-2.0-only
  obj-$(CONFIG_LIVEPATCH) += livepatch.o
 
-livepatch-objs := core.o patch.o shadow.o state.o transition.o
+livepatch-objs := core.o patch.o shadow.o state.o transition.o kprefcount.o
kernel/livepatch/kprefcount.c new file +1 -0
new file mode 100644
index 000000000000..6878033c5ddc
@@ -0,0 +1,89 @@
+#include <linux/init.h>

kernel/livepatch/kprefcount.c line 11

Quoted 10 lines
kernel/livepatch/kprefcount.c new file +10 -0
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/version.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/refcount.h>
+#include <linux/livepatch_refcount.h>
+
+MODULE_LICENSE("GPL");
+

Note that "make W=1" will generate a warning if a module doesn't have a MODULE_DESCRIPTION().

I've been fixing the existing warnings tree-wide and am hoping to prevent new instances from appearing.

One way I've been doing this is by searching lore for patches which add a MODULE_LICENSE() but which do not add a MODULE_DESCRIPTION() since that is a common sign of a missing description.

But in this specific case it seems the MODULE_LICENSE() may be the issue since I don't see how kprefcount.c could ever be built as a module. So in this specific case it seems as if the MODULE_LICENSE() should be removed.

Note that none of the other kernel/livepatch/*.c files have a MODULE_LICENSE(), and CONFIG_LIVEPATCH is a bool and hence does not support being built as a module.

/jeff

#3 lore raw

PATCH 2/2 selftests/livepatch: Add tests for kprefcount_t support

In reply to [email protected]

From: Roman Rashchupkin <[email protected]>

---
 tools/testing/selftests/livepatch/Makefile    |   3 +-
 .../selftests/livepatch/test-kprefcount.sh    |  16 +++
 .../selftests/livepatch/test_modules/Makefile |   4 +-
 .../test_modules/test_klp_kprefcount.c        | 120 ++++++++++++++++++
 .../test_modules/test_klp_refcount.c          |  65 ++++++++++
 5 files changed, 206 insertions(+), 2 deletions(-)
 create mode 100755 tools/testing/selftests/livepatch/test-kprefcount.sh
 create mode 100644 tools/testing/selftests/livepatch/test_modules/test_klp_kprefcount.c
 create mode 100644 tools/testing/selftests/livepatch/test_modules/test_klp_refcount.c
tools/testing/selftests/livepatch/Makefile +2 -1
index 35418a4790be..48926ebc77f2 100644
@@ -10,7 +10,8 @@ TEST_PROGS := \
 	test-state.sh \
 	test-ftrace.sh \
 	test-sysfs.sh \
-	test-syscall.sh
+	test-syscall.sh \
+	test-kprefcount.sh
 
 TEST_FILES := settings
 
tools/testing/selftests/livepatch/test-kprefcount.sh new file +16 -0
new file mode 100755
index 000000000000..8ea6c18f59dd
@@ -0,0 +1,16 @@
+#!/bin/sh
+insmod test_modules/test_klp_refcount.ko
+insmod test_modules/test_klp_kprefcount.ko
+livepatch_enabled=/sys/kernel/livepatch/test_klp_kprefcount/enabled
+while [ ! -e $livepatch_enabled -o $(cat $livepatch_enabled) -eq 0 ]; do
+	sleep 0.01;
+done
+echo 0 > $livepatch_enabled
+while [ $(cat $livepatch_enabled) -eq 1 ]; do
+	sleep 0.01;
+done
+while [ -e $livepatch_enabled ]; do
+	sleep 0.01;
+done
+rmmod test_klp_kprefcount
+rmmod test_klp_refcount
tools/testing/selftests/livepatch/test_modules/Makefile +3 -1
index e6e638c4bcba..c26797372e0d 100644
@@ -11,7 +11,9 @@ obj-m += test_klp_atomic_replace.o \
 	test_klp_state2.o \
 	test_klp_state3.o \
 	test_klp_shadow_vars.o \
-	test_klp_syscall.o
+	test_klp_syscall.o \
+	test_klp_refcount.o \
+	test_klp_kprefcount.o
 
 # Ensure that KDIR exists, otherwise skip the compilation
 modules:
tools/testing/selftests/livepatch/test_modules/test_klp_kprefcount.c new file +120 -0

120 lines not on this page · raw message

tools/testing/selftests/livepatch/test_modules/test_klp_refcount.c new file +65 -0

65 lines not on this page · raw message

Signature · 1 line
2.43.0
maintainer
#4 lore raw
In reply to [email protected]
Quoted 21 lines · On Sun, Jul 14, 2024 at 09:59:32PM +0200, [email protected] wrote:

[PATCH] livepatch: support of modifying refcount_t without underflow after unpatch

CVE fixes sometimes add refcount_inc/dec() pairs to the code with existing refcount_t.
Two problems arise when applying live-patch in this case:
1) After refcount_t is being inc() during system is live-patched, after unpatch the counter value will not be valid, as corresponing dec() would never be called.
2) Underflows are possible in runtime in case dec() is called before corresponding inc() in the live-patched code.

Proposed kprefcount_t functions are using following approach to solve these two problems:
1) In addition to original refcount_t, temporary refcount_t is allocated, and after unpatch it is just removed. This way system is safe with correct refcounting while patch is applied, and no underflow would happend after unpatch.
2) For inc/dec() added by live-patch code, one bit in reference-holder structure is used (unsigned char *ref_holder, kprefholder_flag). In case dec() is called first, it is just ignored as ref_holder bit would still not be initialized.

API is defined include/linux/livepatch_refcount.h:

typedef struct kprefcount_struct {
	refcount_t *refcount;
	refcount_t kprefcount;
	spinlock_t lock;
} kprefcount_t;
Quoted 7 lines

kprefcount_t *kprefcount_alloc(refcount_t *refcount, gfp_t flags);
void kprefcount_free(kprefcount_t *kp_ref);
int kprefcount_read(kprefcount_t *kp_ref);
void kprefcount_inc(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); void kprefcount_dec(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); bool kprefcount_dec_and_test(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);

Hi Roman,

Can you point to a specific upstream commit that this API facilitated a livepatch conversion? That would make a good addition to the Documentation/livepatch/ side of a potential v2.

But first, let me see if I understand the problem correctly. Let's say points A and A' below represent the original kernel code reference get/put pairing in task execution flow. A livepatch adds a new get/put pair, B and B' in the middle like so:

--- execution flow --->
-- A B B' A' -->

There are potential issues if the livepatch is (de)activated mid-sequence, between the new pairings:

problem 1:
-- A . B' A' --> 'B, but no B = extra put!
^ livepatch is activated here

problem 2:
-- A B . A' --> B, but no B' = extra get!
^ livepatch is deactivated here

The first thing that comes to mind is that this might be solved using the existing shadow variable API. When the livepatch takes the new reference (B), it could create a new <struct, NEW_REF> shadow variable instance. The livepatch code to return the reference (B') would then check on the shadow variable existence before doing so. This would solve problem 1.

The second problem is a little trickier. Perhaps the shadow variable approach still works as long as a pre-unpatch hook* were to iterate through all the <*, NEW_REF> shadow variable instances and returned their reference before freeing the shadow variable and declaring the livepatch inactive. I believe that would align the reference counts with original kernel code expectations.

* note this approach probably requires atomic-replace livepatches, so only a single pre-unpatch hook is ever executed.

Also, the proposed patchset looks like it creates a parallel reference counting structure... does this mean that the livepatch will need to update *all* reference counting calls for the API to work (so points A, B, B', and A' in my ascii-art above)? This question loops back to my first point about a real-world example that can be added to Documentation/livepatch/, much like the ones found in the shadow-vars.rst file.

Thanks,

--
Joe

#5 lore raw

Hello.
About upstream commits creating live-patching for which this API would facilitate, I could reference several CVEs:
- CVE-2023-5633
    "drm/vmwgfx: Keep a gem reference to user bos in surfaces"
 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=91398b413d03660fd5828f7b4abc64e884b98069

 drm_gem_object_get(&vbo->tbo.base);/drm_gem_object_put(&tmp_buf->tbo.base);

- CVE-2023-6932
    "ipv4: igmp: fix refcnt uaf issue when receiving igmp query packet"
 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e2b706c69190

    refcount_inc_not_zero(&im->refcnt)/ip_ma_put(im);

- CVE-2022-20566
    "Bluetooth: L2CAP: Fix use-after-free caused by l2cap_chan_put"
 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d0be8347c623e0ac4202a1d4e0373882821f56b0

    kref_get_unless_zero(&c->kref)/l2cap_chan_put(chan)

Only in all of these 3 cases I can remember now, refcount_t is mostly used inside wrapper-functions, and from the top of my head now I don't remember CVEs that plainly add refcount_inc()/dec(). In case the proposed patch is merged, maybe CVE-2023-5633 would be suited best for documentation, or source git could be searched for better example.

Two types of problems that you classify, are exactly what I'm attempting to solve for added refcount_inc/dec() in the code that is added by live-patch. Let's continue with your numbering (1) and (2) for simplicity of discussion.

Concerning problem (1), shadow variables are certainly could be used instead of my refholder bit in reference-holder structures. That's only my preference for simplicity that live-patches code is so often lacking, to use one bit in existing structures instead of hash-table based shadow variables. But certainly shadow-variables are also a good approach, and could be used instead of mine (unsigned char *ref_holder, int kprefholder_flag) in the kprefcount_t API.

About problem (2), iterating through all shadow-variable/refholder instances would also work, but it is just unnecessary processing during unpatch.
In my approach the second kprefcount variable with lifetime of live-patch being applied is used, it provides correct refcounting during live-patch, but the main idea is that this variable can be just safely removed at the unpatch. The only complication could be values of refholder bits, that must be reset at live-patch apply, or probably it is more simple to implement at the unpatch, as all kprefcount_t structs are allocated by patch-code.
---
Roman Rashchupkin

Quoted 85 lines · Joe Lawrence
Quoted 20 lines · On Sun, Jul 14, 2024 at 09:59:32PM +0200, [email protected] wrote:

[PATCH] livepatch: support of modifying refcount_t without underflow after unpatch

CVE fixes sometimes add refcount_inc/dec() pairs to the code with existing refcount_t.
Two problems arise when applying live-patch in this case:
1) After refcount_t is being inc() during system is live-patched, after unpatch the counter value will not be valid, as corresponing dec() would never be called.
2) Underflows are possible in runtime in case dec() is called before corresponding inc() in the live-patched code.

Proposed kprefcount_t functions are using following approach to solve these two problems:
1) In addition to original refcount_t, temporary refcount_t is allocated, and after unpatch it is just removed. This way system is safe with correct refcounting while patch is applied, and no underflow would happend after unpatch.
2) For inc/dec() added by live-patch code, one bit in reference-holder structure is used (unsigned char *ref_holder, kprefholder_flag). In case dec() is called first, it is just ignored as ref_holder bit would still not be initialized.

API is defined include/linux/livepatch_refcount.h:

typedef struct kprefcount_struct {
	refcount_t *refcount;
	refcount_t kprefcount;
	spinlock_t lock;
} kprefcount_t;
Quoted 7 lines

kprefcount_t *kprefcount_alloc(refcount_t *refcount, gfp_t flags);
void kprefcount_free(kprefcount_t *kp_ref);
int kprefcount_read(kprefcount_t *kp_ref);
void kprefcount_inc(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); void kprefcount_dec(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); bool kprefcount_dec_and_test(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);

Hi Roman,

Can you point to a specific upstream commit that this API facilitated a livepatch conversion? That would make a good addition to the Documentation/livepatch/ side of a potential v2.

But first, let me see if I understand the problem correctly. Let's say points A and A' below represent the original kernel code reference get/put pairing in task execution flow. A livepatch adds a new get/put pair, B and B' in the middle like so:

   ---  execution flow  --->
   -- A  B       B'  A'  -->

There are potential issues if the livepatch is (de)activated mid-sequence, between the new pairings:

problem 1:
-- A . B' A' --> 'B, but no B = extra put!
^ livepatch is activated here

problem 2:
-- A B . A' --> B, but no B' = extra get!
^ livepatch is deactivated here

The first thing that comes to mind is that this might be solved using the existing shadow variable API. When the livepatch takes the new reference (B), it could create a new <struct, NEW_REF> shadow variable instance. The livepatch code to return the reference (B') would then check on the shadow variable existence before doing so. This would solve problem 1.

The second problem is a little trickier. Perhaps the shadow variable approach still works as long as a pre-unpatch hook* were to iterate through all the <*, NEW_REF> shadow variable instances and returned their reference before freeing the shadow variable and declaring the livepatch inactive. I believe that would align the reference counts with original kernel code expectations.

* note this approach probably requires atomic-replace livepatches, so only a single pre-unpatch hook is ever executed.

Also, the proposed patchset looks like it creates a parallel reference counting structure... does this mean that the livepatch will need to update *all* reference counting calls for the API to work (so points A, B, B', and A' in my ascii-art above)? This question loops back to my first point about a real-world example that can be added to Documentation/livepatch/, much like the ones found in the shadow-vars.rst file.

Thanks,

--
Joe

#6 lore raw
In reply to Joe Lawrence

Hi all,

Quoted 13 lines · Joe Lawrence
Quoted 1 line · On Sun, Jul 14, 2024 at 09:59:32PM +0200, [email protected] wrote:

But first, let me see if I understand the problem correctly. Let's say points A and A' below represent the original kernel code reference get/put pairing in task execution flow. A livepatch adds a new get/put pair, B and B' in the middle like so:

--- execution flow --->
-- A B B' A' -->

There are potential issues if the livepatch is (de)activated mid-sequence, between the new pairings:

Quoted 7 lines

problem 1:
-- A . B' A' --> 'B, but no B = extra put!
^ livepatch is activated here

problem 2:
-- A B . A' --> B, but no B' = extra get!
^ livepatch is deactivated here

I can confirm that this scenario happens quite often with real world CVE fixes and there's currently no way to implement such changes safely from a livepatch. But I also believe this is an instance of a broader problem class we attempted to solve with that "enhanced" states API proposed and discussed at LPC ([1], there's a link to a recording at the bottom). For reference, see Petr's POC from [2].

Quoted 2 lines

The first thing that comes to mind is that this might be solved using the existing shadow variable API.

Same.

Quoted 11 lines

When the livepatch takes the new
reference (B), it could create a new <struct, NEW_REF> shadow variable instance. The livepatch code to return the reference (B') would then check on the shadow variable existence before doing so. This would solve problem 1.

The second problem is a little trickier. Perhaps the shadow variable approach still works as long as a pre-unpatch hook* were to iterate through all the <*, NEW_REF> shadow variable instances and returned their reference before freeing the shadow variable and declaring the livepatch inactive.

I think the problem of consistently maintaining shadowed reference counts (or anything shadowed for that matter) could be solved with the help of aforementioned states API enhancements, so I would propose to revive Petr's IMO more generic patchset as an alternative.

Thoughts?

Thanks,

Nicolai

[1] https://lpc.events/event/17/contributions/1541/
[2] https://lore.kernel.org/r/[email protected]

Signature · 3 lines
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany
GF: Ivo Totev, Andrew McDonald, Werner Knoblich
(HRB 36809, AG Nürnberg)

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 4 lines
Quoted 2 lines

The first thing that comes to mind is that this might be solved using the existing shadow variable API.

Same.

I just don't have enough experience using live-patch shadow-variables, so I agree that probably that's a better general solution for problem (1) of refcount underflow, than mine refholder flags.

Quoted 6 lines

I can confirm that this scenario happens quite often with real world CVE fixes and there's currently no way to implement such changes safely from a livepatch. But I also believe this is an instance of a broader problem class we attempted to solve with that "enhanced" states API proposed and discussed at LPC ([1], there's a link to a recording at the bottom). For reference, see Petr's POC from [2].

About (2) of incorrect refcount_t value left after unpatch, it seems as a bit different and more special problem with counters, compared to the support of live-patch states, which can be solved for refcount_t in a simpler way.

IMHO adding temporary kprefcount_t variable for the time of live-patch being applied, modifying only this kprefcount_t variable by all added in livepatch inc()/dec(), provides correct refcounting during live-patch is applied. Then at the unpatch this temporaray variable can just safely be discarded. This way the only additional code to live-patch would be functions with original refcount_dec_and_test() calls.

---

Roman Rashchupkin

Quoted 61 lines · Nicolai Stange

Hi all,

Quoted 12 lines · Joe Lawrence

On Sun, Jul 14, 2024 at 09:59:32PM +0200, [email protected] wrote:
But first, let me see if I understand the problem correctly. Let's say points A and A' below represent the original kernel code reference get/put pairing in task execution flow. A livepatch adds a new get/put pair, B and B' in the middle like so:

   ---  execution flow  --->
   -- A  B       B'  A'  -->

There are potential issues if the livepatch is (de)activated mid-sequence, between the new pairings:

Quoted 7 lines

problem 1:
-- A . B' A' --> 'B, but no B = extra put!
^ livepatch is activated here

problem 2:
-- A B . A' --> B, but no B' = extra get!
^ livepatch is deactivated here

I can confirm that this scenario happens quite often with real world CVE fixes and there's currently no way to implement such changes safely from a livepatch. But I also believe this is an instance of a broader problem class we attempted to solve with that "enhanced" states API proposed and discussed at LPC ([1], there's a link to a recording at the bottom). For reference, see Petr's POC from [2].

Quoted 2 lines

The first thing that comes to mind is that this might be solved using the existing shadow variable API.

Same.

Quoted 11 lines

When the livepatch takes the new
reference (B), it could create a new <struct, NEW_REF> shadow variable instance. The livepatch code to return the reference (B') would then check on the shadow variable existence before doing so. This would solve problem 1.

The second problem is a little trickier. Perhaps the shadow variable approach still works as long as a pre-unpatch hook* were to iterate through all the <*, NEW_REF> shadow variable instances and returned their reference before freeing the shadow variable and declaring the livepatch inactive.

I think the problem of consistently maintaining shadowed reference counts (or anything shadowed for that matter) could be solved with the help of aforementioned states API enhancements, so I would propose to revive Petr's IMO more generic patchset as an alternative.

Thoughts?

Thanks,

Nicolai

[1] https://lpc.events/event/17/contributions/1541/
[2] https://lore.kernel.org/r/[email protected]

maintainer
#9 lore raw
Quoted 9 lines · Roman Rashchupkin
Quoted 4 lines
Quoted 2 lines

The first thing that comes to mind is that this might be solved using the existing shadow variable API.

Same.

I just don't have enough experience using live-patch shadow-variables, so I agree that probably that's a better general solution for problem
(1) of refcount underflow, than mine refholder flags.

Yes, a general solution could cover the same problem but for different datatypes, including locks, mutex, etc.

Quoted 6 lines
Quoted 6 lines

I can confirm that this scenario happens quite often with real world CVE fixes and there's currently no way to implement such changes safely from a livepatch. But I also believe this is an instance of a broader problem class we attempted to solve with that "enhanced" states API proposed and discussed at LPC ([1], there's a link to a recording at the bottom). For reference, see Petr's POC from [2].

Thanks for the link -- I thought of that grand-unified shadow/callback/states patch but couldn't find the latest version. (I see that Miroslav has just resurrected it with a fresh review, too.)

Quoted 7 lines
Quoted 7 lines

I think the problem of consistently maintaining shadowed reference counts (or anything shadowed for that matter) could be solved with the help of aforementioned states API enhancements, so I would propose to revive Petr's IMO more generic patchset as an alternative.

Thoughts?

I definitely think the states API enhancement could be used to handle the cases here via shadow variables.

In the meantime, are you using the kprefcount_t API currently via a livepatch support module? i.e. we don't need this in the kernel asap to solve these problems, right?

Signature · 1 line
Joe
maintainer
#10 lore raw
In reply to [email protected]
Quoted 22 lines · On Mon, Jul 15, 2024 at 4:00 AM <[email protected]> wrote:

[PATCH] livepatch: support of modifying refcount_t without underflow after unpatch

CVE fixes sometimes add refcount_inc/dec() pairs to the code with existing refcount_t.
Two problems arise when applying live-patch in this case:
1) After refcount_t is being inc() during system is live-patched, after unpatch the counter value will not be valid, as corresponing dec() would never be called.
2) Underflows are possible in runtime in case dec() is called before corresponding inc() in the live-patched code.

Proposed kprefcount_t functions are using following approach to solve these two problems:
1) In addition to original refcount_t, temporary refcount_t is allocated, and after unpatch it is just removed. This way system is safe with correct refcounting while patch is applied, and no underflow would happend after unpatch.
2) For inc/dec() added by live-patch code, one bit in reference-holder structure is used (unsigned char *ref_holder, kprefholder_flag). In case dec() is called first, it is just ignored as ref_holder bit would still not be initialized.

API is defined include/linux/livepatch_refcount.h:

typedef struct kprefcount_struct {
        refcount_t *refcount;
        refcount_t kprefcount;
        spinlock_t lock;
} kprefcount_t;
Quoted 6 lines

kprefcount_t *kprefcount_alloc(refcount_t *refcount, gfp_t flags);
void kprefcount_free(kprefcount_t *kp_ref);
int kprefcount_read(kprefcount_t *kp_ref);
void kprefcount_inc(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); void kprefcount_dec(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag); bool kprefcount_dec_and_test(kprefcount_t *kp_ref, unsigned char *ref_holder, int kprefholder_flag);

IIUC, kprefcount alone is not enough to solve the two issues. We still need some mechanism to manage the "ref_holder". Shadow variable is probably the best option here.

The primary idea here is to enhance the refcount with a map. This may be too expensive in term memory consumption in some use cases.

Overall, I don't think this change adds much more value on top of shadow variable.

Thanks,
Song