Skip to contentSkip to conversation

reduce maximum memory usage

PATCH Applied Thomas Weißschuh 6 participants 32 messages Dec 27, 2022 – Feb 7, 2023 objtool

lore
Share & tools
lore
#0 lore raw

v2 cover reduce maximum memory usage

The processing of vmlinux.o with objtool is the most memory-intensive step of a kernel build. By reducing the maximum memory usage here we can reduce the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during kernel builds and the build is faster as less swapping has to occur.

To: Josh Poimboeuf <[email protected]>
To: Peter Zijlstra <[email protected]>

---
Changes in v2:
- Warn on out of range values for reloc->type
- Also reduce size of struct special_alt
- Note: v1 did not make it to the lists, only to individual recipients

---
Thomas Weißschuh (8):
objtool: make struct entries[] static and const
objtool: make struct check_options static
objtool: allocate multiple structures with calloc()
objtool: introduce function elf_reloc_set_type
objtool: reduce memory usage of struct reloc
objtool: optimize layout of struct symbol
objtool: optimize layout of struct special_alt
objtool: explicitly cleanup resources on success

 tools/objtool/builtin-check.c           |  4 ++-
 tools/objtool/check.c                   |  6 ++--
 tools/objtool/elf.c                     | 56 +++++++++++++++++++--------------
 tools/objtool/include/objtool/builtin.h |  2 --
 tools/objtool/include/objtool/elf.h     | 13 +++++---
 tools/objtool/include/objtool/special.h |  2 +-
 tools/objtool/special.c                 |  6 ++--
 7 files changed, 51 insertions(+), 38 deletions(-)

---
base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
change-id: 20221216-objtool-memory-06db3b8bf111

Best regards,

Signature · 1 line
Thomas Weißschuh <[email protected]>
#1 lore raw

PATCH v2 1/8 objtool: make struct entries[] static and const

This data is not modified and not used outside of special.c.

Also adapt its users to the constness.

---
 tools/objtool/special.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/objtool/special.c +3 -3
index 9c8d827f69af..baa85c31526b 100644
@@ -26,7 +26,7 @@ struct special_entry {
 	unsigned char key; /* jump_label key */
 };
 
-struct special_entry entries[] = {
+static const struct special_entry entries[] = {
 	{
 		.sec = ".altinstructions",
 		.group = true,
@@ -65,7 +65,7 @@ static void reloc_to_sec_off(struct reloc *reloc, struct section **sec,
 	*off = reloc->sym->offset + reloc->addend;
 }
 
-static int get_alt_entry(struct elf *elf, struct special_entry *entry,
+static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
 			 struct section *sec, int idx,
 			 struct special_alt *alt)
 {
@@ -139,7 +139,7 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
  */
 int special_get_alts(struct elf *elf, struct list_head *alts)
 {
-	struct special_entry *entry;
+	const struct special_entry *entry;
 	struct section *sec;
 	unsigned int nr_entries;
 	struct special_alt *alt;
Signature · 1 line
2.39.0
#2 lore raw

[tip: objtool/core] objtool: Make struct entries[] static and const

The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     4bf285955dcac1810c5677bf69f5ae7a2e9be1cf
Gitweb:        https://git.kernel.org/tip/4bf285955dcac1810c5677bf69f5ae7a2e9be1cf
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:00:57 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Mon, 30 Jan 2023 16:28:18 -08:00

objtool: Make struct entries[] static and const

This data is not modified and not used outside of special.c.

Also adapt its users to the constness.

---
 tools/objtool/special.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/objtool/special.c +3 -3
index 9c8d827..baa85c3 100644
@@ -26,7 +26,7 @@ struct special_entry {
 	unsigned char key; /* jump_label key */
 };
 
-struct special_entry entries[] = {
+static const struct special_entry entries[] = {
 	{
 		.sec = ".altinstructions",
 		.group = true,
@@ -65,7 +65,7 @@ static void reloc_to_sec_off(struct reloc *reloc, struct section **sec,
 	*off = reloc->sym->offset + reloc->addend;
 }
 
-static int get_alt_entry(struct elf *elf, struct special_entry *entry,
+static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
 			 struct section *sec, int idx,
 			 struct special_alt *alt)
 {
@@ -139,7 +139,7 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
  */
 int special_get_alts(struct elf *elf, struct list_head *alts)
 {
-	struct special_entry *entry;
+	const struct special_entry *entry;
 	struct section *sec;
 	unsigned int nr_entries;
 	struct special_alt *alt;
#3 lore raw

[tip: objtool/core] objtool: Make struct entries[] static and const

In reply to Thomas Weißschuh
The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     d93ee0553cf2e83c1696a18423bcf05b94b85e1d
Gitweb:        https://git.kernel.org/tip/d93ee0553cf2e83c1696a18423bcf05b94b85e1d
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:00:57 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Wed, 01 Feb 2023 09:15:22 -08:00

objtool: Make struct entries[] static and const

This data is not modified and not used outside of special.c.

Also adapt its users to the constness.

---
 tools/objtool/special.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/objtool/special.c +3 -3
index 9c8d827..baa85c3 100644
@@ -26,7 +26,7 @@ struct special_entry {
 	unsigned char key; /* jump_label key */
 };
 
-struct special_entry entries[] = {
+static const struct special_entry entries[] = {
 	{
 		.sec = ".altinstructions",
 		.group = true,
@@ -65,7 +65,7 @@ static void reloc_to_sec_off(struct reloc *reloc, struct section **sec,
 	*off = reloc->sym->offset + reloc->addend;
 }
 
-static int get_alt_entry(struct elf *elf, struct special_entry *entry,
+static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
 			 struct section *sec, int idx,
 			 struct special_alt *alt)
 {
@@ -139,7 +139,7 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
  */
 int special_get_alts(struct elf *elf, struct list_head *alts)
 {
-	struct special_entry *entry;
+	const struct special_entry *entry;
 	struct section *sec;
 	unsigned int nr_entries;
 	struct special_alt *alt;
#4 lore raw

PATCH v2 2/8 objtool: make struct check_options static

In reply to Thomas Weißschuh

It is not used outside of builtin-check.c.

Also remove the unused declaration from builtin.h .

---
 tools/objtool/builtin-check.c           | 2 +-
 tools/objtool/include/objtool/builtin.h | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)
tools/objtool/builtin-check.c +1 -1
index a4f39407bf59..7c175198d09f 100644
@@ -65,7 +65,7 @@ static int parse_hacks(const struct option *opt, const char *str, int unset)
 	return found ? 0 : -1;
 }
 
-const struct option check_options[] = {
+static const struct option check_options[] = {
 	OPT_GROUP("Actions:"),
 	OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks),
 	OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"),
tools/objtool/include/objtool/builtin.h +0 -2
index fa45044e3863..2a108e648b7a 100644
@@ -7,8 +7,6 @@
 
 #include <subcmd/parse-options.h>
 
-extern const struct option check_options[];
-
 struct opts {
 	/* actions: */
 	bool dump_orc;
Signature · 1 line
2.39.0
#5 lore raw

[tip: objtool/core] objtool: Make struct check_options static

The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     3b066cdf0893993dcbfabe9ac443e6ff84df6218
Gitweb:        https://git.kernel.org/tip/3b066cdf0893993dcbfabe9ac443e6ff84df6218
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:00:58 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Mon, 30 Jan 2023 16:28:18 -08:00

objtool: Make struct check_options static

It is not used outside of builtin-check.c.

Also remove the unused declaration from builtin.h .

---
 tools/objtool/builtin-check.c           | 2 +-
 tools/objtool/include/objtool/builtin.h | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)
tools/objtool/builtin-check.c +1 -1
index a4f3940..7c17519 100644
@@ -65,7 +65,7 @@ static int parse_hacks(const struct option *opt, const char *str, int unset)
 	return found ? 0 : -1;
 }
 
-const struct option check_options[] = {
+static const struct option check_options[] = {
 	OPT_GROUP("Actions:"),
 	OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks),
 	OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"),
tools/objtool/include/objtool/builtin.h +0 -2
index fa45044..2a108e6 100644
@@ -7,8 +7,6 @@
 
 #include <subcmd/parse-options.h>
 
-extern const struct option check_options[];
-
 struct opts {
 	/* actions: */
 	bool dump_orc;
#6 lore raw

[tip: objtool/core] objtool: Make struct check_options static

In reply to Thomas Weißschuh
The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     cfd66e81799f4a2fdc6447fa99bdb1871f45ff08
Gitweb:        https://git.kernel.org/tip/cfd66e81799f4a2fdc6447fa99bdb1871f45ff08
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:00:58 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Wed, 01 Feb 2023 09:15:23 -08:00

objtool: Make struct check_options static

It is not used outside of builtin-check.c.

Also remove the unused declaration from builtin.h .

---
 tools/objtool/builtin-check.c           | 2 +-
 tools/objtool/include/objtool/builtin.h | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)
tools/objtool/builtin-check.c +1 -1
index a4f3940..7c17519 100644
@@ -65,7 +65,7 @@ static int parse_hacks(const struct option *opt, const char *str, int unset)
 	return found ? 0 : -1;
 }
 
-const struct option check_options[] = {
+static const struct option check_options[] = {
 	OPT_GROUP("Actions:"),
 	OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks),
 	OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"),
tools/objtool/include/objtool/builtin.h +0 -2
index fa45044..2a108e6 100644
@@ -7,8 +7,6 @@
 
 #include <subcmd/parse-options.h>
 
-extern const struct option check_options[];
-
 struct opts {
 	/* actions: */
 	bool dump_orc;
#7 lore raw

PATCH v2 3/8 objtool: allocate multiple structures with calloc()

In reply to Thomas Weißschuh

By using calloc() instead of malloc() in a loop, libc does not have to keep around bookkeeping information for each single structure.

This reduces maximum memory usage while processing vmlinux.o from 3153325 KB to 3035668 KB (-3.7%) on my notebooks "localmodconfig".

---
 tools/objtool/elf.c                 | 42 ++++++++++++++++++-------------------
 tools/objtool/include/objtool/elf.h |  4 ++++
 2 files changed, 25 insertions(+), 21 deletions(-)
tools/objtool/elf.c +21 -21
index 7eae95f33a72..9c326efb8cd9 100644
@@ -285,13 +285,13 @@ static int read_sections(struct elf *elf)
 	    !elf_alloc_hash(section_name, sections_nr))
 		return -1;
 
+	elf->section_data = calloc(sections_nr, sizeof(*sec));
+	if (!elf->section_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < sections_nr; i++) {
-		sec = malloc(sizeof(*sec));
-		if (!sec) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sec, 0, sizeof(*sec));
+		sec = &elf->section_data[i];
 
 		INIT_LIST_HEAD(&sec->symbol_list);
 		INIT_LIST_HEAD(&sec->reloc_list);
@@ -423,13 +423,13 @@ static int read_symbols(struct elf *elf)
 	    !elf_alloc_hash(symbol_name, symbols_nr))
 		return -1;
 
+	elf->symbol_data = calloc(symbols_nr, sizeof(*sym));
+	if (!elf->symbol_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < symbols_nr; i++) {
-		sym = malloc(sizeof(*sym));
-		if (!sym) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sym, 0, sizeof(*sym));
+		sym = &elf->symbol_data[i];
 
 		sym->idx = i;
 
@@ -919,13 +919,13 @@ static int read_relocs(struct elf *elf)
 		sec->base->reloc = sec;
 
 		nr_reloc = 0;
+		sec->reloc_data = calloc(sec->sh.sh_size / sec->sh.sh_entsize, sizeof(*reloc));
+		if (!sec->reloc_data) {
+			perror("calloc");
+			return -1;
+		}
 		for (i = 0; i < sec->sh.sh_size / sec->sh.sh_entsize; i++) {
-			reloc = malloc(sizeof(*reloc));
-			if (!reloc) {
-				perror("malloc");
-				return -1;
-			}
-			memset(reloc, 0, sizeof(*reloc));
+			reloc = &sec->reloc_data[i];
 			switch (sec->sh.sh_type) {
 			case SHT_REL:
 				if (read_rel_reloc(sec, i, reloc, &symndx))
@@ -1458,16 +1458,16 @@ void elf_close(struct elf *elf)
 		list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
 			list_del(&sym->list);
 			hash_del(&sym->hash);
-			free(sym);
 		}
 		list_for_each_entry_safe(reloc, tmpreloc, &sec->reloc_list, list) {
 			list_del(&reloc->list);
 			hash_del(&reloc->hash);
-			free(reloc);
 		}
 		list_del(&sec->list);
-		free(sec);
+		free(sec->reloc_data);
 	}
 
+	free(elf->symbol_data);
+	free(elf->section_data);
 	free(elf);
 }
tools/objtool/include/objtool/elf.h +4 -0
index bb60fd42b46f..1c90f0ac0d53 100644
@@ -39,6 +39,7 @@ struct section {
 	char *name;
 	int idx;
 	bool changed, text, rodata, noinstr, init, truncate;
+	struct reloc *reloc_data;
 };
 
 struct symbol {
@@ -104,6 +105,9 @@ struct elf {
 	struct hlist_head *section_hash;
 	struct hlist_head *section_name_hash;
 	struct hlist_head *reloc_hash;
+
+	struct section *section_data;
+	struct symbol *symbol_data;
 };
 
 #define OFFSET_STRIDE_BITS	4
Signature · 1 line
2.39.0
maintainer
#8 lore raw

Re: [PATCH v2 3/8] objtool: allocate multiple structures with calloc()

Quoted 2 lines · Thomas Weißschuh

By using calloc() instead of malloc() in a loop, libc does not have to keep around bookkeeping information for each single structure.

If we cared about memory leaks, this wouldn't really work because some structures are added in later, after the reading of the original sections and symbols. Luckily we don't care :-)

Signature · 1 line
Josh
#9 lore raw

[tip: objtool/core] objtool: Allocate multiple structures with calloc()

In reply to Thomas Weißschuh
The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     389244804db54977b9316f0dc027ab69b3c1e8bf
Gitweb:        https://git.kernel.org/tip/389244804db54977b9316f0dc027ab69b3c1e8bf
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:00:59 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Mon, 30 Jan 2023 16:28:19 -08:00

objtool: Allocate multiple structures with calloc()

By using calloc() instead of malloc() in a loop, libc does not have to keep around bookkeeping information for each single structure.

This reduces maximum memory usage while processing vmlinux.o from 3153325 KB to 3035668 KB (-3.7%) on my notebooks "localmodconfig".

Note this introduces memory leaks, because some additional structs get added to the lists later after reading the symbols and sections from the original object. Luckily we don't really care about memory leaks in objtool.

---
 tools/objtool/elf.c                 | 42 ++++++++++++++--------------
 tools/objtool/include/objtool/elf.h |  4 +++-
 2 files changed, 25 insertions(+), 21 deletions(-)
tools/objtool/elf.c +21 -21
index 64443a7..6806ce0 100644
@@ -284,13 +284,13 @@ static int read_sections(struct elf *elf)
 	    !elf_alloc_hash(section_name, sections_nr))
 		return -1;
 
+	elf->section_data = calloc(sections_nr, sizeof(*sec));
+	if (!elf->section_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < sections_nr; i++) {
-		sec = malloc(sizeof(*sec));
-		if (!sec) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sec, 0, sizeof(*sec));
+		sec = &elf->section_data[i];
 
 		INIT_LIST_HEAD(&sec->symbol_list);
 		INIT_LIST_HEAD(&sec->reloc_list);
@@ -422,13 +422,13 @@ static int read_symbols(struct elf *elf)
 	    !elf_alloc_hash(symbol_name, symbols_nr))
 		return -1;
 
+	elf->symbol_data = calloc(symbols_nr, sizeof(*sym));
+	if (!elf->symbol_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < symbols_nr; i++) {
-		sym = malloc(sizeof(*sym));
-		if (!sym) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sym, 0, sizeof(*sym));
+		sym = &elf->symbol_data[i];
 
 		sym->idx = i;
 
@@ -918,13 +918,13 @@ static int read_relocs(struct elf *elf)
 		sec->base->reloc = sec;
 
 		nr_reloc = 0;
+		sec->reloc_data = calloc(sec->sh.sh_size / sec->sh.sh_entsize, sizeof(*reloc));
+		if (!sec->reloc_data) {
+			perror("calloc");
+			return -1;
+		}
 		for (i = 0; i < sec->sh.sh_size / sec->sh.sh_entsize; i++) {
-			reloc = malloc(sizeof(*reloc));
-			if (!reloc) {
-				perror("malloc");
-				return -1;
-			}
-			memset(reloc, 0, sizeof(*reloc));
+			reloc = &sec->reloc_data[i];
 			switch (sec->sh.sh_type) {
 			case SHT_REL:
 				if (read_rel_reloc(sec, i, reloc, &symndx))
@@ -1453,16 +1453,16 @@ void elf_close(struct elf *elf)
 		list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
 			list_del(&sym->list);
 			hash_del(&sym->hash);
-			free(sym);
 		}
 		list_for_each_entry_safe(reloc, tmpreloc, &sec->reloc_list, list) {
 			list_del(&reloc->list);
 			hash_del(&reloc->hash);
-			free(reloc);
 		}
 		list_del(&sec->list);
-		free(sec);
+		free(sec->reloc_data);
 	}
 
+	free(elf->symbol_data);
+	free(elf->section_data);
 	free(elf);
 }
tools/objtool/include/objtool/elf.h +4 -0
index bb60fd4..1c90f0a 100644
@@ -39,6 +39,7 @@ struct section {
 	char *name;
 	int idx;
 	bool changed, text, rodata, noinstr, init, truncate;
+	struct reloc *reloc_data;
 };
 
 struct symbol {
@@ -104,6 +105,9 @@ struct elf {
 	struct hlist_head *section_hash;
 	struct hlist_head *section_name_hash;
 	struct hlist_head *reloc_hash;
+
+	struct section *section_data;
+	struct symbol *symbol_data;
 };
 
 #define OFFSET_STRIDE_BITS	4
#10 lore raw

[tip: objtool/core] objtool: Allocate multiple structures with calloc()

In reply to Thomas Weißschuh
The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     8045b8f0b17edf375849f83c80dd05194850b6ed
Gitweb:        https://git.kernel.org/tip/8045b8f0b17edf375849f83c80dd05194850b6ed
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:00:59 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Wed, 01 Feb 2023 09:15:23 -08:00

objtool: Allocate multiple structures with calloc()

By using calloc() instead of malloc() in a loop, libc does not have to keep around bookkeeping information for each single structure.

This reduces maximum memory usage while processing vmlinux.o from 3153325 KB to 3035668 KB (-3.7%) on my notebooks "localmodconfig".

Note this introduces memory leaks, because some additional structs get added to the lists later after reading the symbols and sections from the original object. Luckily we don't really care about memory leaks in objtool.

---
 tools/objtool/elf.c                 | 42 ++++++++++++++--------------
 tools/objtool/include/objtool/elf.h |  4 +++-
 2 files changed, 25 insertions(+), 21 deletions(-)
tools/objtool/elf.c +21 -21
index 64443a7..6806ce0 100644
@@ -284,13 +284,13 @@ static int read_sections(struct elf *elf)
 	    !elf_alloc_hash(section_name, sections_nr))
 		return -1;
 
+	elf->section_data = calloc(sections_nr, sizeof(*sec));
+	if (!elf->section_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < sections_nr; i++) {
-		sec = malloc(sizeof(*sec));
-		if (!sec) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sec, 0, sizeof(*sec));
+		sec = &elf->section_data[i];
 
 		INIT_LIST_HEAD(&sec->symbol_list);
 		INIT_LIST_HEAD(&sec->reloc_list);
@@ -422,13 +422,13 @@ static int read_symbols(struct elf *elf)
 	    !elf_alloc_hash(symbol_name, symbols_nr))
 		return -1;
 
+	elf->symbol_data = calloc(symbols_nr, sizeof(*sym));
+	if (!elf->symbol_data) {
+		perror("calloc");
+		return -1;
+	}
 	for (i = 0; i < symbols_nr; i++) {
-		sym = malloc(sizeof(*sym));
-		if (!sym) {
-			perror("malloc");
-			return -1;
-		}
-		memset(sym, 0, sizeof(*sym));
+		sym = &elf->symbol_data[i];
 
 		sym->idx = i;
 
@@ -918,13 +918,13 @@ static int read_relocs(struct elf *elf)
 		sec->base->reloc = sec;
 
 		nr_reloc = 0;
+		sec->reloc_data = calloc(sec->sh.sh_size / sec->sh.sh_entsize, sizeof(*reloc));
+		if (!sec->reloc_data) {
+			perror("calloc");
+			return -1;
+		}
 		for (i = 0; i < sec->sh.sh_size / sec->sh.sh_entsize; i++) {
-			reloc = malloc(sizeof(*reloc));
-			if (!reloc) {
-				perror("malloc");
-				return -1;
-			}
-			memset(reloc, 0, sizeof(*reloc));
+			reloc = &sec->reloc_data[i];
 			switch (sec->sh.sh_type) {
 			case SHT_REL:
 				if (read_rel_reloc(sec, i, reloc, &symndx))
@@ -1453,16 +1453,16 @@ void elf_close(struct elf *elf)
 		list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
 			list_del(&sym->list);
 			hash_del(&sym->hash);
-			free(sym);
 		}
 		list_for_each_entry_safe(reloc, tmpreloc, &sec->reloc_list, list) {
 			list_del(&reloc->list);
 			hash_del(&reloc->hash);
-			free(reloc);
 		}
 		list_del(&sec->list);
-		free(sec);
+		free(sec->reloc_data);
 	}
 
+	free(elf->symbol_data);
+	free(elf->section_data);
 	free(elf);
 }
tools/objtool/include/objtool/elf.h +4 -0
index bb60fd4..1c90f0a 100644
@@ -39,6 +39,7 @@ struct section {
 	char *name;
 	int idx;
 	bool changed, text, rodata, noinstr, init, truncate;
+	struct reloc *reloc_data;
 };
 
 struct symbol {
@@ -104,6 +105,9 @@ struct elf {
 	struct hlist_head *section_hash;
 	struct hlist_head *section_name_hash;
 	struct hlist_head *reloc_hash;
+
+	struct section *section_data;
+	struct symbol *symbol_data;
 };
 
 #define OFFSET_STRIDE_BITS	4
#11 lore raw

PATCH v2 4/8 objtool: introduce function elf_reloc_set_type

In reply to Thomas Weißschuh

An upcoming patch needs to perform validation when setting reloc->type so introduce a helper to contain this validation.

---
 tools/objtool/check.c               |  6 +++---
 tools/objtool/elf.c                 | 11 ++++++++---
 tools/objtool/include/objtool/elf.h |  2 ++
 3 files changed, 13 insertions(+), 6 deletions(-)
tools/objtool/check.c +3 -3
index 4350be739f4f..971ee6826de7 100644
@@ -1355,7 +1355,7 @@ static void annotate_call_site(struct objtool_file *file,
 	 */
 	if (opts.hack_noinstr && insn->sec->noinstr && sym->profiling_func) {
 		if (reloc) {
-			reloc->type = R_NONE;
+			elf_reloc_set_type(reloc, R_NONE);
 			elf_write_reloc(file->elf, reloc);
 		}
 
@@ -1384,7 +1384,7 @@ static void annotate_call_site(struct objtool_file *file,
 			WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset);
 		if (opts.mnop) {
 			if (reloc) {
-				reloc->type = R_NONE;
+				elf_reloc_set_type(reloc, R_NONE);
 				elf_write_reloc(file->elf, reloc);
 			}
 
@@ -1863,7 +1863,7 @@ static int handle_jump_alt(struct objtool_file *file,
 		struct reloc *reloc = insn_reloc(file, orig_insn);
 
 		if (reloc) {
-			reloc->type = R_NONE;
+			elf_reloc_set_type(reloc, R_NONE);
 			elf_write_reloc(file->elf, reloc);
 		}
 		elf_write_insn(file->elf, orig_insn->sec,
tools/objtool/elf.c +8 -3
index 9c326efb8cd9..ee355beb0d82 100644
@@ -555,7 +555,7 @@ int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offset,
 
 	reloc->sec = sec->reloc;
 	reloc->offset = offset;
-	reloc->type = type;
+	elf_reloc_set_type(reloc, type);
 	reloc->sym = sym;
 	reloc->addend = addend;
 
@@ -872,7 +872,7 @@ static int read_rel_reloc(struct section *sec, int i, struct reloc *reloc, unsig
 		WARN_ELF("gelf_getrel");
 		return -1;
 	}
-	reloc->type = GELF_R_TYPE(reloc->rel.r_info);
+	elf_reloc_set_type(reloc, GELF_R_TYPE(reloc->rel.r_info));
 	reloc->addend = 0;
 	reloc->offset = reloc->rel.r_offset;
 	*symndx = GELF_R_SYM(reloc->rel.r_info);
@@ -885,7 +885,7 @@ static int read_rela_reloc(struct section *sec, int i, struct reloc *reloc, unsi
 		WARN_ELF("gelf_getrela");
 		return -1;
 	}
-	reloc->type = GELF_R_TYPE(reloc->rela.r_info);
+	elf_reloc_set_type(reloc, GELF_R_TYPE(reloc->rela.r_info));
 	reloc->addend = reloc->rela.r_addend;
 	reloc->offset = reloc->rela.r_offset;
 	*symndx = GELF_R_SYM(reloc->rela.r_info);
@@ -1471,3 +1471,8 @@ void elf_close(struct elf *elf)
 	free(elf->section_data);
 	free(elf);
 }
+
+void elf_reloc_set_type(struct reloc *reloc, int type)
+{
+	reloc->type = type;
+}
tools/objtool/include/objtool/elf.h +2 -0
index 1c90f0ac0d53..33ec6cf72325 100644
@@ -83,6 +83,8 @@ struct reloc {
 	bool jump_table_start;
 };
 
+void elf_reloc_set_type(struct reloc *reloc, int type);
+
 #define ELF_HASH_BITS	20
 
 struct elf {
Signature · 1 line
2.39.0
#12 lore raw

PATCH v2 5/8 objtool: reduce memory usage of struct reloc

In reply to Thomas Weißschuh

Use a smaller type for the relocation type and move it to a location in the structure where it avoids wasted padding bytes.

Technically ELF could use up to four bytes for the type. But until now only types up to number 43 have been defined.

Reduce the size of struct reloc on x86_64 from 120 to 112 bytes. This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from 3035668 KB to 2919716 KB (-3.8%) on my notebooks "localmodconfig".

---
 tools/objtool/elf.c                 | 3 +++
 tools/objtool/include/objtool/elf.h | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
tools/objtool/elf.c +3 -0
index ee355beb0d82..182452adaa71 100644
@@ -1474,5 +1474,8 @@ void elf_close(struct elf *elf)
 
 void elf_reloc_set_type(struct reloc *reloc, int type)
 {
+	if (type >= (1U << (8 * sizeof(reloc->type))))
+		WARN("reloc->type out of range: %d", type);
+
 	reloc->type = type;
 }
tools/objtool/include/objtool/elf.h +1 -1
index 33ec6cf72325..2b5becad5a0a 100644
@@ -77,10 +77,10 @@ struct reloc {
 	struct symbol *sym;
 	struct list_head sym_reloc_entry;
 	unsigned long offset;
-	unsigned int type;
 	s64 addend;
 	int idx;
 	bool jump_table_start;
+	unsigned char type;
 };
 
 void elf_reloc_set_type(struct reloc *reloc, int type);
Signature · 1 line
2.39.0
#13 lore raw

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

Hi, Thomas!

Is this likely to cause most RELOC failures? As shown in the following example:

	#include <bfd.h>
	#include <stdio.h>

	int main(void)
	{
		printf("%d\n", BFD_RELOC_S12Z_OPR);
		return 0;
	}

The BFD_RELOC_S12Z_OPR equal to 2366.

Best wishes.
Rong Tao

#14 lore raw

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

Hi Rong,

Quoted 12 lines · Rong Tao

Is this likely to cause most RELOC failures? As shown in the following example:

	#include <bfd.h>
	#include <stdio.h>

	int main(void)
	{
		printf("%d\n", BFD_RELOC_S12Z_OPR);
		return 0;
	}
Quoted 1 line

The BFD_RELOC_S12Z_OPR equal to 2366.

I'm not familiar with libbfd, so please correct me if I'm wrong.

To me BFD_RELOC_S12Z_OPR looks like a value that is only used by libbfd. objtool does not use libbfd.

The only values that objtools should see for the relocation types are the R_* constants from elf.h as they are used by the compiler and linker in the raw elf binary files.

These never exceed the value 255, one byte. Indeed they seem to have been explicitly chosen like this.

$ grep 'define R_.*NUM' /usr/include/elf.h
#define R_68K_NUM	43
#define R_386_NUM	   44
#define R_SPARC_NUM		253
#define R_MIPS_NUM		128
#define R_ALPHA_NUM		46
#define R_ARM_NUM		256
#define R_390_NUM		62
#define R_CRIS_NUM		20
#define R_X86_64_NUM		43
#define R_MN10300_NUM		35
#define R_M32R_NUM		256	/* Keep this the last entry. */
#define R_TILEPRO_NUM		130
#define R_TILEGX_NUM		130
#define R_RISCV_NUM		59

These _NUM constants are the highest actually used values, plus one. So all real values are smaller than 256.

Thomas

Reviewed-by
#15 lore raw

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

Hi, Thomas:

In /usr/include/elf.h has:

#define ELF32_R_TYPE(val) ((val) & 0xff)

#define ELF64_R_TYPE(i)			((i) & 0xffffffff)
                                       ^^^^^^^^^^

So, I still feel that keeping 'unsigned int' is a good option. Can we just use __attribute__((packed)) for wasted padding bytes?

#16 lore raw

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

Hi Rong,

Quoted 10 lines · Rong Tao

Hi, Thomas:

In /usr/include/elf.h has:

#define ELF32_R_TYPE(val) ((val) & 0xff)

#define ELF64_R_TYPE(i)			((i) & 0xffffffff)
                                       ^^^^^^^^^^

So, I still feel that keeping 'unsigned int' is a good option. Can we just use __attribute__((packed)) for wasted padding bytes?

As the struct contains addresses, packing it will create a ton of compiler warnings and will be much more likely to break than reducing the width of 'type'.

Given that reducing the width of 'type'

* is currently absolutely safe,
* is unlikely to break in the future, as the elf designers seem to be actively trying to stay within this range anyways,
* saves 8 bytes from a very heavily allocated struct, having a measurable impact on the build process,

I continue to propose this aproach.

Let's let the maintainers decide.

Thomas

maintainer
#17 lore raw

Re: [PATCH v2 5/8] objtool: reduce memory usage of struct reloc

In reply to Thomas Weißschuh
Quoted 12 lines · Thomas Weißschuh

void elf_reloc_set_type(struct reloc *reloc, int type) {

tools/objtool/elf.c +3 -0
+	if (type >= (1U << (8 * sizeof(reloc->type))))
+		WARN("reloc->type out of range: %d", type);
+
 	reloc->type = type;
 }
tools/objtool/include/objtool/elf.h +0 -0
index 33ec6cf72325..2b5becad5a0a 100644

tools/objtool/include/objtool/elf.h struct reloc { · line 84

Quoted 9 lines
tools/objtool/include/objtool/elf.h +1 -1
@@ -77,10 +77,10 @@ struct reloc {
 	struct symbol *sym;
 	struct list_head sym_reloc_entry;
 	unsigned long offset;
-	unsigned int type;
 	s64 addend;
 	int idx;
 	bool jump_table_start;
+	unsigned char type;
 };

I'd rather not because

a) the helper function isn't very intuitive and we'll probably forget to use it

b) some arches need more than 256 types (see for example aarch64 which objtool may need to support one of these days)

That said, feel free to rearrange the struct fields to at least get some of those bytes back.

Signature · 1 line
Josh
#18 lore raw

PATCH v2 6/8 objtool: optimize layout of struct symbol

In reply to Thomas Weißschuh

Reduce the size of struct symbol on x86_64 from 208 to 200 bytes. This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from 2919716 KB to 2917988 KB (-0.5%) on my notebooks "localmodconfig".

---
 tools/objtool/include/objtool/elf.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
tools/objtool/include/objtool/elf.h +2 -3
index 2b5becad5a0a..f41573ff1f8b 100644
@@ -50,12 +50,11 @@ struct symbol {
 	GElf_Sym sym;
 	struct section *sec;
 	char *name;
-	unsigned int idx;
-	unsigned char bind, type;
+	unsigned int idx, len;
 	unsigned long offset;
-	unsigned int len;
 	unsigned long __subtree_last;
 	struct symbol *pfunc, *cfunc, *alias;
+	unsigned char bind, type;
 	u8 uaccess_safe      : 1;
 	u8 static_call_tramp : 1;
 	u8 retpoline_thunk   : 1;
Signature · 1 line
2.39.0
#19 lore raw

[tip: objtool/core] objtool: Optimize layout of struct symbol

The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     55156d43d62124a92ab80c8f31661609465e257b
Gitweb:        https://git.kernel.org/tip/55156d43d62124a92ab80c8f31661609465e257b
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:01:02 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Mon, 30 Jan 2023 16:28:20 -08:00

objtool: Optimize layout of struct symbol

Reduce the size of struct symbol on x86_64 from 208 to 200 bytes. This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from 2919716 KB to 2917988 KB (-0.5%) on my notebooks "localmodconfig".

---
 tools/objtool/include/objtool/elf.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
tools/objtool/include/objtool/elf.h +2 -3
index 1c90f0a..ad0024d 100644
@@ -50,12 +50,11 @@ struct symbol {
 	GElf_Sym sym;
 	struct section *sec;
 	char *name;
-	unsigned int idx;
-	unsigned char bind, type;
+	unsigned int idx, len;
 	unsigned long offset;
-	unsigned int len;
 	unsigned long __subtree_last;
 	struct symbol *pfunc, *cfunc, *alias;
+	unsigned char bind, type;
 	u8 uaccess_safe      : 1;
 	u8 static_call_tramp : 1;
 	u8 retpoline_thunk   : 1;
#20 lore raw

[tip: objtool/core] objtool: Optimize layout of struct symbol

In reply to Thomas Weißschuh
The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     21a899f9fc78be6b289ee4627bccadf560930eb5
Gitweb:        https://git.kernel.org/tip/21a899f9fc78be6b289ee4627bccadf560930eb5
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:01:02 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Wed, 01 Feb 2023 09:15:24 -08:00

objtool: Optimize layout of struct symbol

Reduce the size of struct symbol on x86_64 from 208 to 200 bytes. This structure is allocated a lot and never freed.

This reduces maximum memory usage while processing vmlinux.o from 2919716 KB to 2917988 KB (-0.5%) on my notebooks "localmodconfig".

---
 tools/objtool/include/objtool/elf.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
tools/objtool/include/objtool/elf.h +2 -3
index 1c90f0a..ad0024d 100644
@@ -50,12 +50,11 @@ struct symbol {
 	GElf_Sym sym;
 	struct section *sec;
 	char *name;
-	unsigned int idx;
-	unsigned char bind, type;
+	unsigned int idx, len;
 	unsigned long offset;
-	unsigned int len;
 	unsigned long __subtree_last;
 	struct symbol *pfunc, *cfunc, *alias;
+	unsigned char bind, type;
 	u8 uaccess_safe      : 1;
 	u8 static_call_tramp : 1;
 	u8 retpoline_thunk   : 1;
#21 lore raw

PATCH v2 7/8 objtool: optimize layout of struct special_alt

In reply to Thomas Weißschuh

Reduce the size of struct special_alt from 72 to 64 bytes.

---
 tools/objtool/include/objtool/special.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/objtool/include/objtool/special.h +1 -1
index dc4721e19002..86d4af9c5aa9 100644
@@ -19,6 +19,7 @@ struct special_alt {
 	bool skip_orig;
 	bool skip_alt;
 	bool jump_or_nop;
+	u8 key_addend;
 
 	struct section *orig_sec;
 	unsigned long orig_off;
@@ -27,7 +28,6 @@ struct special_alt {
 	unsigned long new_off;
 
 	unsigned int orig_len, new_len; /* group only */
-	u8 key_addend;
 };
 
 int special_get_alts(struct elf *elf, struct list_head *alts);
Signature · 1 line
2.39.0
#22 lore raw

[tip: objtool/core] objtool: Optimize layout of struct special_alt

The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     391e6d2abd652b460d9017b2f8a8ded508c09558
Gitweb:        https://git.kernel.org/tip/391e6d2abd652b460d9017b2f8a8ded508c09558
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:01:03 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Mon, 30 Jan 2023 16:28:20 -08:00

objtool: Optimize layout of struct special_alt

Reduce the size of struct special_alt from 72 to 64 bytes.

---
 tools/objtool/include/objtool/special.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/objtool/include/objtool/special.h +1 -1
index dc4721e..86d4af9 100644
@@ -19,6 +19,7 @@ struct special_alt {
 	bool skip_orig;
 	bool skip_alt;
 	bool jump_or_nop;
+	u8 key_addend;
 
 	struct section *orig_sec;
 	unsigned long orig_off;
@@ -27,7 +28,6 @@ struct special_alt {
 	unsigned long new_off;
 
 	unsigned int orig_len, new_len; /* group only */
-	u8 key_addend;
 };
 
 int special_get_alts(struct elf *elf, struct list_head *alts);
#23 lore raw

[tip: objtool/core] objtool: Optimize layout of struct special_alt

In reply to Thomas Weißschuh
The following commit has been merged into the objtool/core branch of tip:

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     a20717aca33b1ff133f513721050fe6c3d7f97b5
Gitweb:        https://git.kernel.org/tip/a20717aca33b1ff133f513721050fe6c3d7f97b5
Author:        Thomas Weißschuh <[email protected]>
AuthorDate:    Tue, 27 Dec 2022 16:01:03 
Committer:     Josh Poimboeuf <[email protected]>
CommitterDate: Wed, 01 Feb 2023 09:15:24 -08:00

objtool: Optimize layout of struct special_alt

Reduce the size of struct special_alt from 72 to 64 bytes.

---
 tools/objtool/include/objtool/special.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/objtool/include/objtool/special.h +1 -1
index dc4721e..86d4af9 100644
@@ -19,6 +19,7 @@ struct special_alt {
 	bool skip_orig;
 	bool skip_alt;
 	bool jump_or_nop;
+	u8 key_addend;
 
 	struct section *orig_sec;
 	unsigned long orig_off;
@@ -27,7 +28,6 @@ struct special_alt {
 	unsigned long new_off;
 
 	unsigned int orig_len, new_len; /* group only */
-	u8 key_addend;
 };
 
 int special_get_alts(struct elf *elf, struct list_head *alts);
#24 lore raw

PATCH v2 8/8 objtool: explicitly cleanup resources on success

In reply to Thomas Weißschuh

Previously the file was only closed and resources properly freed on errors, not on normal exits.

---
 tools/objtool/builtin-check.c | 2 ++
 1 file changed, 2 insertions(+)
tools/objtool/builtin-check.c +2 -0
index 7c175198d09f..e11c766b98ce 100644
@@ -229,5 +229,7 @@ int objtool_run(int argc, const char **argv)
 	if (file->elf->changed)
 		return elf_write(file->elf);
 
+	elf_close(file->elf);
+
 	return 0;
 }
Signature · 1 line
2.39.0
maintainer
#25 lore raw

Re: [PATCH v2 8/8] objtool: explicitly cleanup resources on success

Quoted 13 lines · Thomas Weißschuh

Previously the file was only closed and resources properly freed on errors, not on normal exits.

---
 tools/objtool/builtin-check.c | 2 ++
 1 file changed, 2 insertions(+)
tools/objtool/builtin-check.c +0 -0
index 7c175198d09f..e11c766b98ce 100644

tools/objtool/builtin-check.c int objtool_run(int argc, const char **argv) · line 234

Quoted 6 lines
tools/objtool/builtin-check.c +2 -0
@@ -229,5 +229,7 @@ int objtool_run(int argc, const char **argv)
 	if (file->elf->changed)
 		return elf_write(file->elf);
 
+	elf_close(file->elf);
+
 	return 0;

I'd rather not unless there's a real benefit to doing this, as it will just slow down the exit.

Signature · 1 line
Josh
#26 lore raw

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

In reply to Thomas Weißschuh
Quoted 5 lines · Thomas Weißschuh

The processing of vmlinux.o with objtool is the most memory-intensive step of a kernel build. By reducing the maximum memory usage here we can reduce the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during kernel builds and the build is faster as less swapping has to occur.

Friendly ping.

These patches can also applied one by one, the only dependency is from patch 5 to patch 4.

Thanks,
Thomas

Quoted 37 lines

To: Josh Poimboeuf <[email protected]>
To: Peter Zijlstra <[email protected]>

---
Changes in v2:
- Warn on out of range values for reloc->type
- Also reduce size of struct special_alt
- Note: v1 did not make it to the lists, only to individual recipients

---
Thomas Weißschuh (8):
objtool: make struct entries[] static and const
objtool: make struct check_options static
objtool: allocate multiple structures with calloc()
objtool: introduce function elf_reloc_set_type
objtool: reduce memory usage of struct reloc
objtool: optimize layout of struct symbol
objtool: optimize layout of struct special_alt
objtool: explicitly cleanup resources on success

 tools/objtool/builtin-check.c           |  4 ++-
 tools/objtool/check.c                   |  6 ++--
 tools/objtool/elf.c                     | 56 +++++++++++++++++++--------------
 tools/objtool/include/objtool/builtin.h |  2 --
 tools/objtool/include/objtool/elf.h     | 13 +++++---
 tools/objtool/include/objtool/special.h |  2 +-
 tools/objtool/special.c                 |  6 ++--
 7 files changed, 51 insertions(+), 38 deletions(-)

---
base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
change-id: 20221216-objtool-memory-06db3b8bf111

Best regards,

Signature · 1 line
Thomas Weißschuh <[email protected]>
maintainer
#27 lore raw

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

Quoted 11 lines · Thomas Weißschuh
Quoted 5 lines · Thomas Weißschuh

The processing of vmlinux.o with objtool is the most memory-intensive step of a kernel build. By reducing the maximum memory usage here we can reduce the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during kernel builds and the build is faster as less swapping has to occur.

Friendly ping.

These patches can also applied one by one, the only dependency is from patch 5 to patch 4.

Thanks, I'll go ahead and take five of them now.

Signature · 1 line
Josh
#28 lore raw

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

Quoted 8 lines · Josh Poimboeuf
Quoted 7 lines · Thomas Weißschuh
Quoted 5 lines · Thomas Weißschuh

The processing of vmlinux.o with objtool is the most memory-intensive step of a kernel build. By reducing the maximum memory usage here we can reduce the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during kernel builds and the build is faster as less swapping has to occur.

Quoted 6 lines
Quoted 4 lines

Friendly ping.

These patches can also applied one by one, the only dependency is from patch 5 to patch 4.

Thanks, I'll go ahead and take five of them now.

Thanks.

I have another half-finished series that replaces the doubly-linked list_heads used by elf.h with a custom singly-linked list. This would save a few pointers per struct.

Do you think this is worth it?

Thomas

maintainer
#29 lore raw

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

Quoted 16 lines · Thomas Weißschuh
Quoted 8 lines · Josh Poimboeuf
Quoted 7 lines · Thomas Weißschuh
Quoted 5 lines · Thomas Weißschuh

The processing of vmlinux.o with objtool is the most memory-intensive step of a kernel build. By reducing the maximum memory usage here we can reduce the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during kernel builds and the build is faster as less swapping has to occur.

Quoted 6 lines
Quoted 4 lines

Friendly ping.

These patches can also applied one by one, the only dependency is from patch 5 to patch 4.

Thanks, I'll go ahead and take five of them now.

Quoted 7 lines

Thanks.

I have another half-finished series that replaces the doubly-linked list_heads used by elf.h with a custom singly-linked list. This would save a few pointers per struct.

Do you think this is worth it?

Maybe, depending on the memory savings.

Signature · 1 line
Josh
maintainer
#30 lore raw

Re: [PATCH v2 0/8] objtool: reduce maximum memory usage

Quoted 19 lines · Josh Poimboeuf
Quoted 18 lines · Thomas Weißschuh
Quoted 8 lines · Josh Poimboeuf
Quoted 7 lines · Thomas Weißschuh
Quoted 5 lines · Thomas Weißschuh

The processing of vmlinux.o with objtool is the most memory-intensive step of a kernel build. By reducing the maximum memory usage here we can reduce the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during kernel builds and the build is faster as less swapping has to occur.

Quoted 6 lines
Quoted 4 lines

Friendly ping.

These patches can also applied one by one, the only dependency is from patch 5 to patch 4.

Thanks, I'll go ahead and take five of them now.

Thanks.

Quoted 7 lines
Quoted 5 lines

I have another half-finished series that replaces the doubly-linked list_heads used by elf.h with a custom singly-linked list. This would save a few pointers per struct.

Do you think this is worth it?

Maybe, depending on the memory savings.

FYI, there were more memory usage complaints, so Peter worked up a nice series to do this and more:

https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=objtool/core

Signature · 1 line
Josh
#31 lore raw

RE: [PATCH v2 0/8] objtool: reduce maximum memory usage

In reply to Thomas Weißschuh

From: Thomas Weißschuh.

Quoted 1 line

Sent: 31 January 2023 03:55

...

Quoted 5 lines

I have another half-finished series that replaces the doubly-linked list_heads used by elf.h with a custom singly-linked list. This would save a few pointers per struct.

Do you think this is worth it?

If you allocate the structures in blocks of (say) 256 you can use an array of pointers to the blocks and then a 32bit index instead of a 64bit pointer.

For real space-saving you might decide that the index can never exceed 2^^24 and use a bitfield!

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)