Skip to contentSkip to conversation

Implement MTE tag compression for swapped pages

PATCH In review Alexander Potapenko 3 participants 13 messages Nov 13, 2023 – Dec 14, 2023

Subsystems (2)

v9 of 12 In review Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v9 cover Implement MTE tag compression for swapped pages

Currently, when MTE pages are swapped out, the tags are kept in the memory, occupying PAGE_SIZE/32 bytes per page. This is especially problematic for devices that use zram-backed in-memory swap, because tags stored uncompressed in the heap effectively reduce the available amount of swap memory.

The RLE-based algorithm suggested by Evgenii Stepanov and implemented in this patch series is able to efficiently compress fixed-size tag buffers, resulting in practical compression ratio of 2x. In many cases it is possible to store the compressed data in 63-bit Xarray values, resulting in no extra memory allocations.

This patch series depends on "lib/bitmap: add bitmap_{read,write}()"
(https://lore.kernel.org/linux-arm-kernel/[email protected]/T/)
that is mailed separately.

v9:
- split off the stats collection code into a separate patch in the series (as suggested by Yury Norov)

v8:
- split off the bitmap_read()/bitmap_write() series
- simplified the compression logic (only compress data if it fits into a pointer)

v7:
- fixed comments by Yury Norov, Andy Shevchenko, Rasmus Villemoes
- added perf tests for bitmap_read()/bitmap_write()
- more efficient bitmap_write() implementation (meant to be sent in v5)

v6:
- fixed comments by Yury Norov
- fixed handling of sizes divisible by MTE_GRANULES_PER_PAGE / 2 (caught while testing on a real device)

v5:
- fixed comments by Andy Shevchenko, Catalin Marinas, and Yury Norov
- added support for 16K- and 64K pages
- more efficient bitmap_write() implementation

v4:
- fixed a bunch of comments by Andy Shevchenko and Yury Norov
- added Documentation/arch/arm64/mte-tag-compression.rst

v3:
- as suggested by Andy Shevchenko, use bitmap_get_value()/bitmap_set_value() written by Syed Nayyar Waris
- switched to unsigned long to reduce typecasts
- simplified the compression code

v2:
- as suggested by Yuri Norov, replace the poorly implemented struct bitq with <linux/bitmap.h>

Alexander Potapenko (4):
arm64: mte: implement CONFIG_ARM64_MTE_COMP
arm64: mte: add a test for MTE tags compression
arm64: mte: add compression support to mteswap.c
arm64: mte: implement CONFIG_ARM64_MTE_SWAP_STATS

 Documentation/arch/arm64/index.rst            |   1 +
 .../arch/arm64/mte-tag-compression.rst        | 166 ++++++++
 arch/arm64/Kconfig                            |  37 ++
 arch/arm64/include/asm/mtecomp.h              |  39 ++
 arch/arm64/mm/Makefile                        |   2 +
 arch/arm64/mm/mtecomp.c                       | 257 +++++++++++++
 arch/arm64/mm/mtecomp.h                       |  12 +
 arch/arm64/mm/mteswap.c                       | 110 +++++-
 arch/arm64/mm/test_mtecomp.c                  | 364 ++++++++++++++++++
 9 files changed, 985 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/arch/arm64/mte-tag-compression.rst
 create mode 100644 arch/arm64/include/asm/mtecomp.h
 create mode 100644 arch/arm64/mm/mtecomp.c
 create mode 100644 arch/arm64/mm/mtecomp.h
 create mode 100644 arch/arm64/mm/test_mtecomp.c
Signature · 1 line
2.42.0.869.gea05f2083d-goog
#1 lore raw

PATCH v9 1/4 arm64: mte: implement CONFIG_ARM64_MTE_COMP

The config implements the algorithm compressing memory tags for ARM MTE during swapping.

The algorithm is based on RLE and specifically targets buffers of tags corresponding to a single page. In many cases a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation.

---
v8:
- As suggested by Catalin Marinas, only compress tags if they can be stored inline. This simplifies the code drastically.
- Update the documentation.
- Split off patches introducing bitmap_read()/bitmap_write().

v6:
- shuffle bits in inline handles so that they can't be confused with

    canonical pointers;

- use kmem_cache_zalloc() to allocate compressed storage
- correctly handle range size overflow
- minor documentation fixes, clarify the special cases

v5:
- make code PAGE_SIZE-agnostic, remove hardcoded constants, updated the docs
- implement debugfs interface
- Address comments by Andy Shevchenko:
- update description of mtecomp.c
- remove redundant assignments, simplify mte_tags_to_ranges()
- various code simplifications
- introduce mtecomp.h
- add :export: to Documentation/arch/arm64/mte-tag-compression.rst

v4:
- Addressed comments by Andy Shevchenko:
- expanded "MTE" to "Memory Tagging Extension" in Kconfig
- fixed kernel-doc comments, moved them to C source
- changed variables to unsigned where applicable
- some code simplifications, fewer unnecessary assignments
- added the mte_largest_idx_bits() helper
- added namespace prefixes to all functions
- added missing headers (but removed bits.h)
- Addressed comments by Yury Norov:
- removed test-only functions from mtecomp.h
- dropped the algoritm name (all functions are now prefixed with "mte")
- added more comments
- got rid of MTE_RANGES_INLINE
- renamed bitmap_{get,set}_value() to bitmap_{read,write}()
- moved the big comment explaining the algorithm to Documentation/arch/arm64/mte-tag-compression.rst, expanded it, add a link to it from Documentation/arch/arm64/index.rst
- removed hardcoded ranges from mte_alloc_size()/mte_size_to_ranges()

v3:
- Addressed comments by Andy Shevchenko:
- use bitmap_{set,get}_value() writte by Syed Nayyar Waris
- switched to unsigned long everywhere (fewer casts)
- simplified the code, removed redundant checks
- dropped ea0_compress_inline()
- added bit size constants and helpers to access the bitmap
- explicitly initialize all compressed sizes in ea0_compress_to_buf()
- initialize all handle bits

v2:
- as suggested by Yury Norov, switched from struct bitq (which is not needed anymore) to <linux/bitmap.h>
- add missing symbol exports

---
 Documentation/arch/arm64/index.rst            |   1 +
 .../arch/arm64/mte-tag-compression.rst        | 154 +++++++++++
 arch/arm64/Kconfig                            |  11 +
 arch/arm64/include/asm/mtecomp.h              |  39 +++
 arch/arm64/mm/Makefile                        |   1 +
 arch/arm64/mm/mtecomp.c                       | 257 ++++++++++++++++++
 arch/arm64/mm/mtecomp.h                       |  12 +
 7 files changed, 475 insertions(+)
 create mode 100644 Documentation/arch/arm64/mte-tag-compression.rst
 create mode 100644 arch/arm64/include/asm/mtecomp.h
 create mode 100644 arch/arm64/mm/mtecomp.c
 create mode 100644 arch/arm64/mm/mtecomp.h
Documentation/arch/arm64/index.rst +1 -0
index d08e924204bf1..bf6c1583233a9 100644
@@ -19,6 +19,7 @@ ARM64 Architecture
     legacy_instructions
     memory
     memory-tagging-extension
+    mte-tag-compression
     perf
     pointer-authentication
     ptdump
Documentation/arch/arm64/mte-tag-compression.rst new file +154 -0

154 lines not on this page · raw message

arch/arm64/Kconfig +11 -0
index 78f20e6327120..c2fbe8e492e00 100644
@@ -2073,6 +2073,17 @@ config ARM64_EPAN
 	  if the cpu does not implement the feature.
 endmenu # "ARMv8.7 architectural features"
 
+config ARM64_MTE_COMP
+	bool "Tag compression for ARM64 Memory Tagging Extension"
+	default y
+	depends on ARM64_MTE
+	help
+	  Enable tag compression support for ARM64 Memory Tagging Extension.
+
+	  Tag buffers corresponding to swapped RAM pages are compressed using
+	  RLE to conserve heap memory. In the common case compressed tags
+	  occupy 2.5x less memory.
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
arch/arm64/include/asm/mtecomp.h new file +39 -0
new file mode 100644
index 0000000000000..b9a3a921a38d4
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __ASM_MTECOMP_H
+#define __ASM_MTECOMP_H
+
+#include <linux/types.h>
+
+/**
+ * mte_is_compressed() - check if the supplied pointer contains compressed tags.
+ * @ptr: pointer returned by kmalloc() or mte_compress().
+ *
+ * Returns: true iff bit 0 of @ptr is 1, which is only possible if @ptr was
+ * returned by mte_is_compressed().
+ */
+static inline bool mte_is_compressed(void *ptr)
+{
+	return ((unsigned long)ptr & 1);
+}
+
+#if defined(CONFIG_ARM64_MTE_COMP)
+
+void *mte_compress(u8 *tags);
+bool mte_decompress(void *handle, u8 *tags);
+
+#else
+
+static inline void *mte_compress(u8 *tags)
+{
+	return NULL;
+}
+
+static inline bool mte_decompress(void *data, u8 *tags)
+{
+	return false;
+}
+
+#endif // CONFIG_ARM64_MTE_COMP
+
+#endif // __ASM_MTECOMP_H
arch/arm64/mm/Makefile +1 -0
index dbd1bc95967d0..46778f6dd83c2 100644
@@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE)	+= trans_pgd.o
 obj-$(CONFIG_TRANS_TABLE)	+= trans_pgd-asm.o
 obj-$(CONFIG_DEBUG_VIRTUAL)	+= physaddr.o
 obj-$(CONFIG_ARM64_MTE)		+= mteswap.o
+obj-$(CONFIG_ARM64_MTE_COMP)	+= mtecomp.o
 KASAN_SANITIZE_physaddr.o	+= n
 
 obj-$(CONFIG_KASAN)		+= kasan_init.o
arch/arm64/mm/mtecomp.c new file +257 -0

257 lines not on this page · raw message

arch/arm64/mm/mtecomp.h new file +12 -0
new file mode 100644
index 0000000000000..b94cf0384f2af
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef ARCH_ARM64_MM_MTECOMP_H_
+#define ARCH_ARM64_MM_MTECOMP_H_
+
+/* Functions exported from mtecomp.c for test_mtecomp.c. */
+void mte_tags_to_ranges(u8 *tags, u8 *out_tags, unsigned short *out_sizes,
+			size_t *out_len);
+void mte_ranges_to_tags(u8 *r_tags, unsigned short *r_sizes, size_t r_len,
+			u8 *tags);
+
+#endif  // ARCH_ARM64_MM_TEST_MTECOMP_H_
Signature · 1 line
2.42.0.869.gea05f2083d-goog
maintainer Acked-by on v9 1/4
#2 lore raw

Re: [PATCH v9 1/4] arm64: mte: implement CONFIG_ARM64_MTE_COMP

Quoted 10 lines · Alexander Potapenko

The config implements the algorithm compressing memory tags for ARM MTE during swapping.

The algorithm is based on RLE and specifically targets buffers of tags corresponding to a single page. In many cases a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation.

Quoted 3 lines
arch/arm64/mm/mtecomp.c new file +3 -0
+void mte_tags_to_ranges(u8 *tags, u8 *out_tags, unsigned short *out_sizes,
+			size_t *out_len)
+{

[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+}
+EXPORT_SYMBOL_NS(mte_tags_to_ranges, MTECOMP);

I may have asked before, what the reason for EXPORT_SYMBOL_NS? Is it for the kunit tests? Otherwise we don't expect those to be accessed from modules.

Signature · 1 line
Catalin
maintainer
#3 lore raw

Re: [PATCH v9 1/4] arm64: mte: implement CONFIG_ARM64_MTE_COMP

Quoted 18 lines · Catalin Marinas
Quoted 10 lines · Alexander Potapenko

The config implements the algorithm compressing memory tags for ARM MTE during swapping.

The algorithm is based on RLE and specifically targets buffers of tags corresponding to a single page. In many cases a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation.

Quoted 3 lines
arch/arm64/mm/mtecomp.c new file +3 -0
+void mte_tags_to_ranges(u8 *tags, u8 *out_tags, unsigned short *out_sizes,
+			size_t *out_len)
+{

[...]

Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+}
+EXPORT_SYMBOL_NS(mte_tags_to_ranges, MTECOMP);

I may have asked before, what the reason for EXPORT_SYMBOL_NS? Is it for the kunit tests? Otherwise we don't expect those to be accessed from modules.

It looks like it. The Kconfig option for the test is tristate and it calls this symbol directly.

Will

#4 lore raw

Re: [PATCH v9 1/4] arm64: mte: implement CONFIG_ARM64_MTE_COMP

Quoted 27 lines · Will Deacon
Quoted 18 lines · Catalin Marinas
Quoted 10 lines · Alexander Potapenko

The config implements the algorithm compressing memory tags for ARM MTE during swapping.

The algorithm is based on RLE and specifically targets buffers of tags corresponding to a single page. In many cases a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation.

Quoted 3 lines
arch/arm64/mm/mtecomp.c new file +3 -0
+void mte_tags_to_ranges(u8 *tags, u8 *out_tags, unsigned short *out_sizes,
+                   size_t *out_len)
+{

[...]

Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+}
+EXPORT_SYMBOL_NS(mte_tags_to_ranges, MTECOMP);

I may have asked before, what the reason for EXPORT_SYMBOL_NS? Is it for the kunit tests? Otherwise we don't expect those to be accessed from modules.

Quoted 4 lines

It looks like it. The Kconfig option for the test is tristate and it calls this symbol directly.

Will

What Will said.

#5 lore raw

PATCH v9 2/4 arm64: mte: add a test for MTE tags compression

In reply to Alexander Potapenko

Ensure that tag sequences containing alternating values are compressed to buffers of expected size and correctly decompressed afterwards.

---
v9:
- minor changes to Kconfig description

v8:
- adapt to the simplified compression algorithm

v6:
- add test_decompress_invalid() to ensure invalid handles are ignored;
- add test_upper_bits(), which is a regression test for a case where

    an inline handle looked like an out-of-line one;
  - add test_compress_nonzero() to ensure a full nonzero tag array is
    compressed correctly;

- add test_two_ranges() to test cases when the input buffer is divided into two ranges.

v5:
- remove hardcoded constants, added test setup/teardown;
- support 16- and 64K pages;
- replace nested if-clauses with expected_size_from_ranges();
- call mte_release_handle() after tests that perform

    compression/decompression;

- address comments by Andy Shevchenko:
- fix include order;
- use mtecomp.h instead of function prototypes.

v4:
- addressed comments by Andy Shevchenko:
- expanded MTE to "Memory Tagging Extension" in Kconfig
- changed signed variables to unsigned where applicable
- added missing header dependencies

- addressed comments by Yury Norov:
- moved test-only declarations from mtecomp.h into this test
- switched to the new "mte"-prefixed function names, dropped the mentions of "EA0"
- added test_tag_to_ranges_n()

v3:
- addressed comments by Andy Shevchenko in another patch:
- switched from u64 to unsigned long
- added MODULE_IMPORT_NS(MTECOMP)
- fixed includes order

---
 arch/arm64/Kconfig           |  11 ++
 arch/arm64/mm/Makefile       |   1 +
 arch/arm64/mm/test_mtecomp.c | 364 +++++++++++++++++++++++++++++++++++
 3 files changed, 376 insertions(+)
 create mode 100644 arch/arm64/mm/test_mtecomp.c
arch/arm64/Kconfig +11 -0
index c2fbe8e492e00..92104994aaf18 100644
@@ -2084,6 +2084,17 @@ config ARM64_MTE_COMP
 	  RLE to conserve heap memory. In the common case compressed tags
 	  occupy 2.5x less memory.
 
+config ARM64_MTE_COMP_KUNIT_TEST
+	tristate "Test tag compression for ARM64 Memory Tagging Extension" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	depends on KUNIT && ARM64_MTE_COMP
+	help
+	  Test MTE compression algorithm enabled by CONFIG_ARM64_MTE_COMP.
+
+	  Ensure that certain tag sequences containing alternating values can
+	  be compressed into pointer-size values and correctly decompressed
+	  afterwards.
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
arch/arm64/mm/Makefile +1 -0
index 46778f6dd83c2..170dc62b010b9 100644
@@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE)	+= trans_pgd-asm.o
 obj-$(CONFIG_DEBUG_VIRTUAL)	+= physaddr.o
 obj-$(CONFIG_ARM64_MTE)		+= mteswap.o
 obj-$(CONFIG_ARM64_MTE_COMP)	+= mtecomp.o
+obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) += test_mtecomp.o
 KASAN_SANITIZE_physaddr.o	+= n
 
 obj-$(CONFIG_KASAN)		+= kasan_init.o
arch/arm64/mm/test_mtecomp.c new file +364 -0

364 lines not on this page · raw message

Signature · 1 line
2.42.0.869.gea05f2083d-goog
#7 lore raw

PATCH v9 3/4 arm64: mte: add compression support to mteswap.c

In reply to Alexander Potapenko

Update mteswap.c to perform inline compression of memory tags when possible.

If CONFIG_ARM64_MTE_COMP is enabled, mteswap.c will attempt to compress saved tags for a struct page and store them directly in Xarray entry instead of wasting heap space.

Soon after booting Android, tag compression saves ~2x memory previously spent by mteswap.c on tag allocations. On a moderately loaded device with ~20% tagged pages, this leads to saving several megabytes of kernel heap:

  # cat /sys/kernel/debug/mteswap/stats
  8 bytes:	102496 allocations,	67302 deallocations
  128 bytes:	212234 allocations,	178278 deallocations
  uncompressed tag storage size:	8851200
  compressed tag storage size:	4346368

(statistics collection is introduced in the following patch)

---
v9:
- as requested by Yury Norov, split off statistics collection into a separate patch
- minor fixes

v8:
- adapt to the new compression API, abandon mteswap_{no,}comp.c
- move stats collection to mteswap.c

v5:
- drop a dead variable from _mte_free_saved_tags() in mteswap_comp.c
- ensure MTE compression works with arbitrary page sizes
- update patch description

v4:
- minor code simplifications suggested by Andy Shevchenko, added missing header dependencies
- changed compression API names to reflect modifications made to memcomp.h (as suggested by Yury Norov)

v3:
- Addressed comments by Andy Shevchenko in another patch:
- fixed includes order
- replaced u64 with unsigned long
- added MODULE_IMPORT_NS(MTECOMP)

---
 arch/arm64/mm/mteswap.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
arch/arm64/mm/mteswap.c +17 -2
index a31833e3ddc54..70f5c8ecd640d 100644
@@ -6,6 +6,8 @@
 #include <linux/swap.h>
 #include <linux/swapops.h>
 #include <asm/mte.h>
+#include <asm/mtecomp.h>
+#include "mtecomp.h"
 
 static DEFINE_XARRAY(mte_pages);
 
@@ -17,12 +19,13 @@ void *mte_allocate_tag_storage(void)
 
 void mte_free_tag_storage(char *storage)
 {
-	kfree(storage);
+	if (!mte_is_compressed(storage))
+		kfree(storage);
 }
 
 int mte_save_tags(struct page *page)
 {
-	void *tag_storage, *ret;
+	void *tag_storage, *compressed_storage, *ret;
 
 	if (!page_mte_tagged(page))
 		return 0;
@@ -32,6 +35,11 @@ int mte_save_tags(struct page *page)
 		return -ENOMEM;
 
 	mte_save_page_tags(page_address(page), tag_storage);
+	compressed_storage = mte_compress(tag_storage);
+	if (compressed_storage) {
+		mte_free_tag_storage(tag_storage);
+		tag_storage = compressed_storage;
+	}
 
 	/* lookup the swap entry.val from the page */
 	ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage,
@@ -50,13 +58,20 @@ int mte_save_tags(struct page *page)
 void mte_restore_tags(swp_entry_t entry, struct page *page)
 {
 	void *tags = xa_load(&mte_pages, entry.val);
+	void *tag_storage = NULL;
 
 	if (!tags)
 		return;
 
 	if (try_page_mte_tagging(page)) {
+		if (mte_is_compressed(tags)) {
+			tag_storage = mte_allocate_tag_storage();
+			mte_decompress(tags, tag_storage);
+			tags = tag_storage;
+		}
 		mte_restore_page_tags(page_address(page), tags);
 		set_page_mte_tagged(page);
+		mte_free_tag_storage(tag_storage);
 	}
 }
 
Signature · 1 line
2.42.0.869.gea05f2083d-goog
maintainer Reviewed-by on v9 3/4
#8 lore raw

Re: [PATCH v9 3/4] arm64: mte: add compression support to mteswap.c

Quoted 17 lines · Alexander Potapenko

Update mteswap.c to perform inline compression of memory tags when possible.

If CONFIG_ARM64_MTE_COMP is enabled, mteswap.c will attempt to compress saved tags for a struct page and store them directly in Xarray entry instead of wasting heap space.

Soon after booting Android, tag compression saves ~2x memory previously spent by mteswap.c on tag allocations. On a moderately loaded device with ~20% tagged pages, this leads to saving several megabytes of kernel heap:

  # cat /sys/kernel/debug/mteswap/stats
  8 bytes:	102496 allocations,	67302 deallocations
  128 bytes:	212234 allocations,	178278 deallocations
  uncompressed tag storage size:	8851200
  compressed tag storage size:	4346368
Quoted 3 lines

(statistics collection is introduced in the following patch)

#9 lore raw

PATCH v9 4/4 arm64: mte: implement CONFIG_ARM64_MTE_SWAP_STATS

In reply to Alexander Potapenko

Provide a config to collect the usage statistics for ARM MTE tag compression. This patch introduces allocation/deallocation counters for buffers that were stored uncompressed (and thus occupy 128 bytes of heap plus the Xarray overhead to store a pointer) and those that were compressed into 8-byte pointers (effectively using 0 bytes of heap in addition to the Xarray overhead).

The counters are exposed to the userspace via
/sys/kernel/debug/mteswap/stats:

  # cat /sys/kernel/debug/mteswap/stats
  8 bytes:      102496 allocations,     67302 deallocations
  128 bytes:    212234 allocations,     178278 deallocations
  uncompressed tag storage size:        8851200
  compressed tag storage size:  4346368

---

This patch was split off from the "arm64: mte: add compression support to mteswap.c" patch
(https://lore.kernel.org/linux-arm-kernel/ZUVulBKVYK7cq2rJ@yury-ThinkPad/T/#m819ec30beb9de53d5c442f7e3247456f8966d88a)

v9:
- add this patch, put the stats behind a separate config, mention /sys/kernel/debug/mteswap/stats in the documentation

---
 .../arch/arm64/mte-tag-compression.rst        | 12 +++
 arch/arm64/Kconfig                            | 15 +++
 arch/arm64/mm/mteswap.c                       | 93 ++++++++++++++++++-
 3 files changed, 118 insertions(+), 2 deletions(-)
Documentation/arch/arm64/mte-tag-compression.rst +12 -0
index 8fe6b51a9db6d..4c25b96f7d4b5 100644
@@ -145,6 +145,18 @@ Tag compression and decompression implicitly rely on the fixed MTE tag size
 (4 bits) and number of tags per page. Should these values change, the algorithm
 may need to be revised.
 
+Stats
+=====
+
+When `CONFIG_ARM64_MTE_SWAP_STATS` is enabled, `arch/arm64/mm/mteswap.c` exports
+usage statistics for tag compression used when swapping tagged pages. The data
+can be accessed via debugfs::
+
+  # cat /sys/kernel/debug/mteswap/stats
+  8 bytes:        10438 allocations,      10417 deallocations
+  128 bytes:      26180 allocations,      26179 deallocations
+  uncompressed tag storage size:  2816
+  compressed tag storage size:    128
 
 Programming Interface
 =====================
arch/arm64/Kconfig +15 -0
index 92104994aaf18..60c969bd4078b 100644
@@ -2095,6 +2095,21 @@ config ARM64_MTE_COMP_KUNIT_TEST
 	  be compressed into pointer-size values and correctly decompressed
 	  afterwards.
 
+config ARM64_MTE_SWAP_STATS
+	bool "Collect usage statistics of tag compression for swapped MTE tags"
+	default y
+	depends on ARM64_MTE && ARM64_MTE_COMP
+	help
+	  Collect usage statistics for ARM64 MTE tag compression during swapping.
+
+	  Adds allocation/deallocation counters for buffers that were stored
+	  uncompressed (and thus occupy 128 bytes of heap plus the Xarray
+	  overhead to store a pointer) and those that were compressed into
+	  8-byte pointers (effectively using 0 bytes of heap in addition to
+	  the Xarray overhead).
+	  The counters are exposed to the userspace via
+	  /sys/kernel/debug/mteswap/stats.
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
arch/arm64/mm/mteswap.c +91 -2

121 lines not on this page · raw message

Signature · 1 line
2.42.0.869.gea05f2083d-goog
maintainer Acked-by on v9 4/4
#10 lore raw

Re: [PATCH v9 4/4] arm64: mte: implement CONFIG_ARM64_MTE_SWAP_STATS

Quoted 10 lines · Alexander Potapenko

Provide a config to collect the usage statistics for ARM MTE tag compression. This patch introduces allocation/deallocation counters for buffers that were stored uncompressed (and thus occupy 128 bytes of heap plus the Xarray overhead to store a pointer) and those that were compressed into 8-byte pointers (effectively using 0 bytes of heap in addition to the Xarray overhead).

The counters are exposed to the userspace via
/sys/kernel/debug/mteswap/stats:

Quoted 8 lines
  # cat /sys/kernel/debug/mteswap/stats
  8 bytes:      102496 allocations,     67302 deallocations
  128 bytes:    212234 allocations,     178278 deallocations
  uncompressed tag storage size:        8851200
  compressed tag storage size:  4346368
maintainer
#11 lore raw
In reply to Alexander Potapenko
Quoted 12 lines · Alexander Potapenko

Currently, when MTE pages are swapped out, the tags are kept in the memory, occupying PAGE_SIZE/32 bytes per page. This is especially problematic for devices that use zram-backed in-memory swap, because tags stored uncompressed in the heap effectively reduce the available amount of swap memory.

The RLE-based algorithm suggested by Evgenii Stepanov and implemented in this patch series is able to efficiently compress fixed-size tag buffers, resulting in practical compression ratio of 2x. In many cases it is possible to store the compressed data in 63-bit Xarray values, resulting in no extra memory allocations.

Quoted 3 lines

This patch series depends on "lib/bitmap: add bitmap_{read,write}()"
(https://lore.kernel.org/linux-arm-kernel/[email protected]/T/)
that is mailed separately.

That's a shame, because it means I can't apply the series as-is:

arch/arm64/mm/mtecomp.c: In function ‘mte_bitmap_write’:
arch/arm64/mm/mtecomp.c:105:2: error: implicit declaration of function ‘bitmap_write’; did you mean ‘bitmap_free’? [-Werror=implicit-function-declaration]

  105 |  bitmap_write(bitmap, value, *pos, bits);
      |  ^~~~~~~~~~~~
      |  bitmap_free
arch/arm64/mm/mtecomp.c: In function ‘mte_bitmap_read’:
arch/arm64/mm/mtecomp.c:198:9: error: implicit declaration of function ‘bitmap_read’; did you mean ‘bitmap_remap’? [-Werror=implicit-function-declaration]
  198 |  return bitmap_read(bitmap, start, bits);
      |         ^~~~~~~~~~~
      |         bitmap_remap

cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:243: arch/arm64/mm/mtecomp.o] Error 1

Do you really have such a hard dependency on those new bitmap ops?

Will

#12 lore raw
Quoted 14 lines · Will Deacon
Quoted 12 lines · Alexander Potapenko

Currently, when MTE pages are swapped out, the tags are kept in the memory, occupying PAGE_SIZE/32 bytes per page. This is especially problematic for devices that use zram-backed in-memory swap, because tags stored uncompressed in the heap effectively reduce the available amount of swap memory.

The RLE-based algorithm suggested by Evgenii Stepanov and implemented in this patch series is able to efficiently compress fixed-size tag buffers, resulting in practical compression ratio of 2x. In many cases it is possible to store the compressed data in 63-bit Xarray values, resulting in no extra memory allocations.

Quoted 6 lines
Quoted 3 lines

This patch series depends on "lib/bitmap: add bitmap_{read,write}()"
(https://lore.kernel.org/linux-arm-kernel/[email protected]/T/)
that is mailed separately.

That's a shame, because it means I can't apply the series as-is:

Uh-oh, sorry about that. There was another series depending on the bitmap_read/bitmap_write API, and I thought mailing those patches separately would speed things up.
But in fact Yury requested them to have at least one user, so now that the MTE series is also acked I'd better bring everything back together.
Let me send out another version.

Quoted 11 lines

arch/arm64/mm/mtecomp.c: In function ‘mte_bitmap_write’:
arch/arm64/mm/mtecomp.c:105:2: error: implicit declaration of function ‘bitmap_write’; did you mean ‘bitmap_free’? [-Werror=implicit-function-declaration]

  105 |  bitmap_write(bitmap, value, *pos, bits);
      |  ^~~~~~~~~~~~
      |  bitmap_free
arch/arm64/mm/mtecomp.c: In function ‘mte_bitmap_read’:
arch/arm64/mm/mtecomp.c:198:9: error: implicit declaration of function ‘bitmap_read’; did you mean ‘bitmap_remap’? [-Werror=implicit-function-declaration]
  198 |  return bitmap_read(bitmap, start, bits);
      |         ^~~~~~~~~~~
      |         bitmap_remap
Quoted 5 lines

cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:243: arch/arm64/mm/mtecomp.o] Error 1

Do you really have such a hard dependency on those new bitmap ops?

Having them in bitmap.h seems natural - this way they can be reused by other people.