Skip to contentSkip to conversation

makedumpfile security key filtering with eppic

PATCH Superseded Aravinda Prasad 5 participants 44 messages Nov 8, 2012 – Dec 10, 2012

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover makedumpfile security key filtering with eppic

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Eppic is an interpreter that facilitates access to the symbol and type information stored in an executable image or a dump file. Eppic defines a language semantic which is similar to C. Eppic macros can be used to specify rules/commands to erase data in an image file. makedumpfile will interpret the rules/commands provided by eppic macros with the help of eppic library and will suitably erase the required data in a dump file. Eppic provides a lot of language constructs like conditional statements, logical and arithmetic operators, nested loops, functions, etc., to traverse nested lists and trees and conditionally erase data in the dump file, enabling users to literally erase any data in the dump file which is accessible through global symbols.

The series of patches integrates eppic with makdumpfile. These patches require eppic library libeppic.a and eppic_api.h header file. The libeppic.a library can be built from the eppic source code available at the following URL:

http://code.google.com/p/eppic/

TODO:

- Currently, works only for symbols in vmlinux, extend it to module symbols
- Functionality support:
- Implement the following callback functions.
- apialignment
- apigetenum
- apigetdefs
- Other functionalities specified in the code with TODO tag
- Support specifying eppic macros in makedumpfile.conf file
- Update erase info

Changelog from v1 to v2:

- Re-based to v1.5.0
- Introduced EPPIC=on in makefile, and hence eppic is now optional
- Incorporated review comments from Atsushi
- Minor formatting changes

Regards,
Aravinda
---

Aravinda Prasad (7):
Initialize and setup eppic
makedumpfile and eppic interface layer
Eppic call back functions to query a dump image Implement apigetctype call back function Implement apimember and apigetrtype call back functions Extend eppic built-in functions to include memset function Support fully typed symbol access mode

 Makefile          |    7 +
 dwarf_info.c      |  367 +++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |   18 ++
 erase_info.c      |   83 +++++++++-
 erase_info.h      |    5 +
 extension_eppic.c |  451 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 extension_eppic.h |   78 +++++++++
 makedumpfile.c    |    7 +
 makedumpfile.h    |    6 +
 9 files changed, 1018 insertions(+), 4 deletions(-)
 create mode 100644 extension_eppic.c
 create mode 100644 extension_eppic.h
Signature · 7 lines
Aravinda Prasad


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

PATCH v2 1/7 Initialize and setup eppic

This patch contains routines which initialize eppic and register call back function which will be called whenever a new eppic macro is loaded using eppic_load() API. The registered call back function executes the eppic macro as soon as it is loaded.

The -ltinfo is included in LIBS in the Makefile, because eppic currently calls few functions in libtinfo and the compiler complains about it, if not included. I think the paths where libeppic calls libtinfo functions are not exercised by makedumpfile.

Including -ltinfo will increase the size of makedumpfile static version

TODO:
- Check out a way to get rid of -ltinfo

---
 Makefile          |    7 ++++
 extension_eppic.c |   84 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 extension_eppic.h |   23 +++++++++++++++
 3 files changed, 114 insertions(+), 0 deletions(-)
 create mode 100644 extension_eppic.c
 create mode 100644 extension_eppic.h
Makefile +7 -0
index d5fc09e..3ac8e63 100644
@@ -55,6 +55,13 @@ LIBS := -llzo2 $(LIBS)
 CFLAGS += -DUSELZO
 endif
 
+ifeq ($(EPPIC), on)
+LIBS := -leppic -ltinfo $(LIBS)
+CFLAGS += -DEPPIC
+SRC_PART += extension_eppic.c
+OBJ_PART += extension_eppic.o
+endif
+
 all: makedumpfile
 
 $(OBJ_PART): $(SRC_PART)
extension_eppic.c new file +84 -0

84 lines not on this page · raw message

extension_eppic.h new file +23 -0
new file mode 100644
index 0000000..ca74ce4
@@ -0,0 +1,23 @@
+/*
+ * extension_eppic.h
+ *
+ * Created by: Aravinda Prasad <[email protected]>
+ *
+ * Copyright (C) 2012  IBM Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+#ifndef _EXTENSION_EPPIC_H
+#define _EXTENSION_EPPIC_H
+
+#include "eppic_api.h"
+
+#endif /* _EXTENSION_EPPIC_H */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

Re: [PATCH v2 1/7] Initialize and setup eppic

On Thu, Nov 08, 2012 at 07:08:04PM +0530, Aravinda Prasad wrote:

[..]

Makefile LIBS := -llzo2 $(LIBS) · line 59

Quoted 10 lines
Makefile +2 -0
index d5fc09e..3ac8e63 100644
@@ -55,6 +55,13 @@ LIBS := -llzo2 $(LIBS)
 CFLAGS += -DUSELZO
 endif
 
+ifeq ($(EPPIC), on)
+LIBS := -leppic -ltinfo $(LIBS)

So you also pull in libarary "tinfo". Does that 1MB number include this too? What's the total size bloat for makedumpfile once we compile with EPPIC on.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

Re: [PATCH v2 1/7] Initialize and setup eppic

Quoted 11 lines · Vivek Goyal

On Thu, Nov 08, 2012 at 07:08:04PM +0530, Aravinda Prasad wrote:

[..]

Quoted 8 lines
Makefile +0 -0
index d5fc09e..3ac8e63 100644
@@ -55,6 +55,13 @@ LIBS := -llzo2 $(LIBS)
 CFLAGS += -DUSELZO
 endif
 
Quoted 6 lines

Makefile LIBS := -llzo2 $(LIBS) · line 59

Quoted 2 lines
Makefile +2 -0
+ifeq ($(EPPIC), on)
+LIBS := -leppic -ltinfo $(LIBS)

So you also pull in libarary "tinfo". Does that 1MB number include this too? What's the total size bloat for makedumpfile once we compile with EPPIC on.

Yes it includes. The actual size bloat for makedumpfile with EPPIC=on is around 850KB

Quoted 4 lines

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#4 lore raw

PATCH v2 2/7 makedumpfile and eppic interface layer

In reply to Aravinda Prasad

This patch extends the makedumpfile functionality to include eppic language support. The patch initializes eppic and then loads and executes the specified eppic macros.

This patch also includes "--eppic" option to makedumpfile command to specify the eppic macro or the directory containing eppic macros. The specified eppic macro will be executed to scrub the data in the dumpfile. In case of a directory, all the eppic macros inside the directory will be executed for scrubbing the data.

TODO
- Support specifying eppic macros in the makedumpfile.conf file. A new command should be added to makedumpfile.conf to identify an eppic macro. This command could be used in any section of the makedumpfile.conf, where the section name indicates the kernel module name. makedumpfile will only search for symbols encountered by eppic macro only in the specified module. Because, searching these symbols in all the modules will cause huge performance overhead.

---
 erase_info.c      |   50 ++++++++++++++++++++++++++++++++++++++++++++++++--
 erase_info.h      |    4 ++++
 extension_eppic.h |    3 +++
 makedumpfile.c    |    7 ++++++-
 makedumpfile.h    |    6 ++++++
 5 files changed, 67 insertions(+), 3 deletions(-)
erase_info.c +48 -2
index e2e6a52..5d4cfd1 100644
@@ -1811,6 +1811,34 @@ process_config_file(const char *name_config)
 	return TRUE;
 }
 
+
+/* Process the eppic macro using eppic library */
+#ifdef EPPIC
+static int
+process_eppic_file(char *name_config)
+{
+	int ret = 0;
+
+	/* Initialize eppic*/
+	ret = eppic_init();
+	if (ret) {
+		ERRMSG("Eppic initialization failed\n");
+		return FALSE;
+	}
+
+	/* TODO
+	 * Support specifying eppic macros in makedumpfile.conf file
+	 */
+
+	/* Load/compile and execute the eppic macro */
+	eppic_load(name_config);
+
+	eppic_unload(name_config);
+
+	return TRUE;
+}
+#endif
+
 static void
 split_filter_info(struct filter_info *prev, unsigned long long next_paddr,
 						size_t size)
@@ -1904,7 +1932,7 @@ extract_filter_info(unsigned long long start_paddr,
 int
 gather_filter_info(void)
 {
-	int ret;
+	int ret = TRUE;
 
 	/*
 	 * Before processing filter config file, load the symbol data of
@@ -1915,7 +1943,25 @@ gather_filter_info(void)
 	if (!load_module_symbols())
 		return FALSE;
 
-	ret = process_config_file(info->name_filterconfig);
+	/*
+	 * This patch supports specifying both makedumpfile.conf and
+	 * eppic macro at the same time. Whether to retain or discard the
+	 * functionality provided by makedumpfile.conf is open for
+	 * discussion
+	 */
+	if (info->name_filterconfig)
+		ret = process_config_file(info->name_filterconfig);
+
+	if (info->name_eppic_config)
+#ifdef EPPIC
+		ret &= process_eppic_file(info->name_eppic_config);
+#else
+	{
+		MSG("'-eppic' option is disabled, because this ");
+		MSG("binary doesn't support scrubbing using eppic.\n");
+		MSG("Try `make EPPIC=on` when building.\n");
+	}
+#endif
 
 	/*
 	 * Remove modules symbol information, we dont need now.
erase_info.h +4 -0
index 636fcfa..0e3c3a3 100644
@@ -19,6 +19,10 @@
 #ifndef _ERASE_INFO_H
 #define _ERASE_INFO_H
 
+#ifdef EPPIC
+#include "extension_eppic.h"
+#endif
+
 #define MAX_SIZE_STR_LEN (26)
 
 /*
extension_eppic.h +3 -0
index ca74ce4..67efd11 100644
@@ -20,4 +20,7 @@
 
 #include "eppic_api.h"
 
+int eppic_init(void);    /* Eppic initialize */
+
 #endif /* _EXTENSION_EPPIC_H */
+
makedumpfile.c +6 -1
index 4931319..3ba74ca 100644
@@ -6943,7 +6943,8 @@ retry:
 		}
 	}
 
-	if (info->name_filterconfig && !gather_filter_info())
+	if ((info->name_filterconfig || info->name_eppic_config)
+			&& !gather_filter_info())
 		return FALSE;
 
 	if (!create_dump_bitmap())
@@ -7779,6 +7780,7 @@ static struct option longopts[] = {
 	{"diskset", required_argument, NULL, 'k'},
 	{"non-cyclic", no_argument, NULL, 'Y'},
 	{"cyclic-buffer", required_argument, NULL, 'Z'},
+	{"eppic", required_argument, NULL, 'S'},
 	{0, 0, 0, 0}
 };
 
@@ -7874,6 +7876,9 @@ main(int argc, char *argv[])
 		case 's':
 			info->flag_split = 1;
 			break;
+		case 'S':
+			info->name_eppic_config = optarg;
+			break;
 		case 'r':
 			info->flag_reassemble = 1;
 			break;
makedumpfile.h +6 -0
index 2b88fa4..f7e0616 100644
@@ -830,6 +830,12 @@ struct DumpInfo {
 	FILE		*file_filterconfig;
 
 	/*
+	 * Filter config file containing eppic language filtering rules
+	 * to filter out kernel data from vmcore
+	 */
+	char		*name_eppic_config;
+
+	/*
 	 * diskdimp info:
 	 */
 	int		block_order;

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

PATCH v2 3/7 Eppic call back functions to query a dump image

In reply to Aravinda Prasad

This patch implements a series of apigetuint* call back functions which are used to access data from the dump image. Eppic uses these call back functions to fetch the actual value of the global variables.

This patch also adds other call back functions as a place holder which will be implemented in later patches. This is mainly to avoid compilation error while registering the call back functions using eppic_apiset() function.

---
 extension_eppic.c |  128 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 extension_eppic.h |   50 ++++++++++++++++++++-
 2 files changed, 176 insertions(+), 2 deletions(-)
extension_eppic.c +127 -1
index fc47b1f..26c945f 100644
@@ -20,6 +20,7 @@
 #include <sys/types.h>
 #include <fcntl.h>
 
+#include "makedumpfile.h"
 #include "extension_eppic.h"
 
 /*
@@ -64,6 +65,131 @@ reg_callback(char *name, int load)
 	return;
 }
 
+/*
+ * Call back functions for eppic to query the dump image
+ */
+
+static int
+apigetmem(ull iaddr, void *p, int nbytes)
+{
+	return readmem(VADDR, iaddr, p, nbytes);
+}
+
+static int
+apiputmem(ull iaddr, void *p, int nbytes)
+{
+	return 1;
+}
+
+static char *
+apimember(char *mname, ull pidx, type_t *tm,
+		member_t *m, ull *lidx)
+{
+	return 0;
+}
+
+static int
+apigetctype(int ctype, char *name, type_t *tout)
+{
+	return 0;
+}
+
+static char *
+apigetrtype(ull idx, type_t *t)
+{
+	return "";
+}
+
+static int
+apialignment(ull idx)
+{
+	return 0;
+}
+
+int
+apigetval(char *name, ull *val, VALUE_S *value)
+{
+	ull ptr = 0;
+
+	ptr = get_symbol_addr(name);
+	if (!ptr)
+		return 0;
+
+	*val = ptr;
+	return 1;
+}
+
+static enum_t *
+apigetenum(char *name)
+{
+	return 0;
+}
+
+static def_t *
+apigetdefs(void)
+{
+	return 0;
+}
+
+static uint8_t
+apigetuint8(void *ptr)
+{
+	uint8_t val;
+	if (!readmem(VADDR, (unsigned long)ptr, (char *)&val, sizeof(val)))
+		return (uint8_t) -1;
+	return val;
+}
+
+static uint16_t
+apigetuint16(void *ptr)
+{
+	uint16_t val;
+	if (!readmem(VADDR, (unsigned long)ptr, (char *)&val, sizeof(val)))
+		return (uint16_t) -1;
+	return val;
+}
+
+static uint32_t
+apigetuint32(void *ptr)
+{
+	uint32_t val;
+	if (!readmem(VADDR, (unsigned long)ptr, (char *)&val, sizeof(val)))
+		return (uint32_t) -1;
+	return val;
+}
+
+static uint64_t
+apigetuint64(void *ptr)
+{
+	uint64_t val;
+	if (!readmem(VADDR, (unsigned long)ptr, (char *)&val, sizeof(val)))
+		return (uint64_t) -1;
+	return val;
+}
+
+static char *
+apifindsym(char *p)
+{
+	return NULL;
+}
+
+apiops icops = {
+	apigetmem,
+	apiputmem,
+	apimember,
+	apigetctype,
+	apigetrtype,
+	apialignment,
+	apigetval,
+	apigetenum,
+	apigetdefs,
+	apigetuint8,
+	apigetuint16,
+	apigetuint32,
+	apigetuint64,
+	apifindsym
+};
+
 
 /* Initialize eppic */
 int
@@ -72,7 +198,7 @@ eppic_init()
 	if (eppic_open() >= 0) {
 
 		/* Register call back functions */
-		eppic_apiset(NULL, 3, sizeof(long), 0);
+		eppic_apiset(&icops, 3, sizeof(long), 0);
 
 		/* set the new function callback */
 		eppic_setcallback(reg_callback);
extension_eppic.h +49 -1

54 lines not on this page · raw message

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

PATCH v2 4/7 Implement apigetctype call back function

In reply to Aravinda Prasad

libeppic will call apigetctype call back function whenever it encounters a token in the eppic macro. The call back function will use DWARF to query information related to the requested token and will pass it back to eppic using libeppic API calls. If the token does not exist, then apigetctype call returns 0.

---
 dwarf_info.c      |  109 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |    9 ++++
 extension_eppic.c |   28 +++++++++++++-
 3 files changed, 145 insertions(+), 1 deletions(-)
dwarf_info.c +109 -0
index fb11e49..ae2db19 100644
@@ -51,6 +51,7 @@ struct dwarf_info {
 	long	enum_number;		/* OUT */
 	unsigned char	type_flag;	/* OUT */
 	char	src_name[LEN_SRCFILE];	/* OUT */
+	Dwarf_Off die_offset;		/* OUT */
 };
 static struct dwarf_info	dwarf_info;
 
@@ -102,6 +103,22 @@ is_search_typedef(int cmd)
 }
 
 static int
+is_search_domain(int cmd)
+{
+	if ((cmd == DWARF_INFO_GET_DOMAIN_STRUCT)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_TYPEDEF)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_ARRAY)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_UNION)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_ENUM)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_REF)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_STRING)
+		|| (cmd == DWARF_INFO_GET_DOMAIN_BASE))
+		return TRUE;
+	else
+		return FALSE;
+}
+
+static int
 process_module (Dwfl_Module *dwflmod,
 		void **userdata __attribute__ ((unused)),
 		const char *name __attribute__ ((unused)),
@@ -736,6 +753,74 @@ search_symbol(Dwarf_Die *die, int *found)
 }
 
 static void
+search_domain(Dwarf_Die *die, int *found)
+{
+	int tag;
+	const char *name;
+	short flag = 0;
+	Dwarf_Die die_type;
+
+	do {
+		tag  = dwarf_tag(die);
+		name = dwarf_diename(die);
+
+		/*
+		 * Descend into anonymous members and search for the
+		 * needed domain there.
+		 */
+		if (!name) {
+			if (!get_die_type(die, &die_type))
+				continue;
+
+			if (is_anonymous_container(&die_type)) {
+				Dwarf_Die child;
+
+				if (dwarf_child(&die_type, &child) != 0)
+					continue;
+
+				search_domain(&child, found);
+
+				if (*found)
+					return;
+			}
+		}
+
+		if ((!name) || strcmp(name, dwarf_info.symbol_name))
+			continue;
+
+		switch (dwarf_info.cmd) {
+		case DWARF_INFO_GET_DOMAIN_STRUCT:
+			if (tag == DW_TAG_structure_type)
+				flag = 1;
+			break;
+		case DWARF_INFO_GET_DOMAIN_UNION:
+			if (tag == DW_TAG_union_type)
+				flag = 1;
+			break;
+		case DWARF_INFO_GET_DOMAIN_TYPEDEF:
+			if (tag == DW_TAG_typedef)
+				flag = 1;
+			break;
+		/* TODO
+		 * Implement functionality for the rest of the domains
+		 */
+		}
+
+		if (!flag)
+			continue;
+
+		dwarf_info.struct_size = dwarf_bytesize(die);
+
+		if (dwarf_info.struct_size > 0) {
+			if (found)
+				*found = TRUE;
+			dwarf_info.die_offset = dwarf_dieoffset(die);
+			break;
+		}
+	} while (!dwarf_siblingof(die, die));
+}
+
+static void
 search_die_tree(Dwarf_Die *die, int *found)
 {
 	Dwarf_Die child;
@@ -760,6 +845,9 @@ search_die_tree(Dwarf_Die *die, int *found)
 
 	else if (is_search_typedef(dwarf_info.cmd))
 		search_typedef(die, found);
+
+	else if (is_search_domain(dwarf_info.cmd))
+		search_domain(die, found);
 }
 
 static int
@@ -1145,6 +1233,27 @@ get_source_filename(char *structname, char *src_name, int cmd)
 }
 
 /*
+ * Get the domain information of the symbol
+ */
+long
+get_domain(char *symname, int cmd, unsigned long long *die)
+{
+	dwarf_info.cmd         = cmd;
+	dwarf_info.symbol_name = symname;
+	dwarf_info.type_name   = NULL;
+	dwarf_info.struct_size = NOT_FOUND_STRUCTURE;
+	dwarf_info.die_offset  = 0;
+
+	if (!get_debug_info())
+		return 0;
+
+	if (die)
+		*die = (unsigned long long) dwarf_info.die_offset;
+
+	return dwarf_info.struct_size;
+}
+
+/*
  * Set the dwarf_info with kernel/module debuginfo file information.
  */
 int
dwarf_info.h +9 -0
index 8d0084d..94d75ea 100644
@@ -46,6 +46,14 @@ enum {
 	DWARF_INFO_CHECK_SYMBOL_ARRAY_TYPE,
 	DWARF_INFO_GET_SYMBOL_TYPE,
 	DWARF_INFO_GET_MEMBER_TYPE,
+	DWARF_INFO_GET_DOMAIN_STRUCT,
+	DWARF_INFO_GET_DOMAIN_TYPEDEF,
+	DWARF_INFO_GET_DOMAIN_ARRAY,
+	DWARF_INFO_GET_DOMAIN_UNION,
+	DWARF_INFO_GET_DOMAIN_ENUM,
+	DWARF_INFO_GET_DOMAIN_REF,
+	DWARF_INFO_GET_DOMAIN_STRING,
+	DWARF_INFO_GET_DOMAIN_BASE,
 };
 
 char *get_dwarf_module_name(void);
@@ -60,6 +68,7 @@ char *get_member_type_name(char *structname, char *membername, int cmd, long *si
 long get_array_length(char *name01, char *name02, unsigned int cmd);
 long get_enum_number(char *enum_name);
 int get_source_filename(char *structname, char *src_name, int cmd);
+long get_domain(char *symname, int cmd, unsigned long long *die);
 int set_dwarf_debuginfo(char *mod_name, char *os_release, char *name_debuginfo, int fd_debuginfo);
 
 #endif  /* DWARF_INFO_H */
extension_eppic.c +27 -1
index 26c945f..6e9fff8 100644
@@ -91,7 +91,33 @@ apimember(char *mname, ull pidx, type_t *tm,
 static int
 apigetctype(int ctype, char *name, type_t *tout)
 {
-	return 0;
+	long size = 0;
+	unsigned long long die = 0;
+
+	switch (ctype) {
+	case V_TYPEDEF:
+		size = get_domain(name, DWARF_INFO_GET_DOMAIN_TYPEDEF, &die);
+		break;
+	case V_STRUCT:
+		size = get_domain(name, DWARF_INFO_GET_DOMAIN_STRUCT, &die);
+		break;
+	case V_UNION:
+		size = get_domain(name, DWARF_INFO_GET_DOMAIN_UNION, &die);
+		break;
+	/* TODO
+	 * Implement for all the domains
+	 */
+	}
+
+	if (size <= 0 || !die)
+		return 0;
+
+	/* populate */
+	eppic_type_settype(tout, ctype);
+	eppic_type_setsize(tout, size);
+	eppic_type_setidx(tout, (ull)(unsigned long)die);
+	eppic_pushref(tout, 0);
+	return 1;
 }
 
 static char *

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw

PATCH v2 5/7 Implement apimember and apigetrtype call back functions

In reply to Aravinda Prasad

The patch includes functionality for apimember and apigetrtype eppic callback routines along with helper functions to fetch information related to the member of the structure/union

Whenever a structure/union member is accessed inside the eppic macro, eppic will query makedumpfile through call back functions requesting more information on the structure or union member. The information includes member name, offset from structure, data type and size. makedumpfile will get all these information using DWARF and pass back to eppic using libeppic API calls.

---
 dwarf_info.c      |  206 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |    7 ++
 extension_eppic.c |  179 +++++++++++++++++++++++++++++++++++++++++++++-
 extension_eppic.h |    2 +
 4 files changed, 390 insertions(+), 4 deletions(-)
dwarf_info.c +206 -0
index ae2db19..7d07c41 100644
@@ -463,6 +463,29 @@ get_dwarf_base_type(Dwarf_Die *die)
 	return TRUE;
 }
 
+/*
+ * Get the die, given the offset
+ */
+static int
+get_die_from_offset(Dwarf_Off offset, Dwarf_Die *die)
+{
+	if (!init_dwarf_info())
+		return FALSE;
+
+	if ((!offset) || (!die))
+		return FALSE;
+
+	if (!dwarf_offdie(dwarf_info.dwarfd, offset, die)) {
+		ERRMSG("Can't find the DIE.\n");
+		return FALSE;
+	}
+
+	return TRUE;
+}
+
+/*
+ * Function for searching struct page.union.struct.mapping.
+ */
 static int
 is_anonymous_container(Dwarf_Die *die)
 {
@@ -1254,6 +1277,189 @@ get_domain(char *symname, int cmd, unsigned long long *die)
 }
 
 /*
+ * Get the number of fields in a structure or union provided the
+ * die offset of the structure or union
+ */
+int
+get_die_nfields(unsigned long long die_off)
+{
+	int tag, nfields = 0;
+	Dwarf_Die result, child, *die;
+
+	if (!get_die_from_offset((Dwarf_Off) die_off, &result)) {
+		ERRMSG("Can't find the DIE.\n");
+		return -1;
+	}
+
+	die = &result;
+	tag = dwarf_tag(die);
+	if (tag != DW_TAG_structure_type && tag != DW_TAG_union_type) {
+		ERRMSG("DIE is not of structure or union type.\n");
+		return -1;
+	}
+
+	if (dwarf_child(die, &child) != 0)
+		return -1;
+
+	/* Find the number of fields in the structure */
+	die = &child;
+	do {
+		tag = dwarf_tag(die);
+		if (tag == DW_TAG_member)
+			nfields++;
+		else
+			continue;
+	} while (!dwarf_siblingof(die, die));
+
+	return nfields;
+}
+
+/*
+ * Get the information of the structure member given by index
+ */
+int
+get_die_member(unsigned long long die_off, int index, long *offset,
+		char **name, int *nbits, int *fbits, unsigned long long *m_die)
+{
+	int tag, size, nfields = 0;
+	Dwarf_Die result, child, die_base, *die;
+
+	if (!offset || !nbits || !fbits || !name || !m_die)
+		return -1;
+
+	if (!get_die_from_offset((Dwarf_Off) die_off, &result)) {
+		ERRMSG("Can't find the DIE.\n");
+		return -1;
+	}
+
+	die = &result;
+	tag = dwarf_tag(die);
+	if (tag != DW_TAG_structure_type && tag != DW_TAG_union_type) {
+		ERRMSG("DIE is not of structure or union type.\n");
+		return -1;
+	}
+
+	if (dwarf_child(die, &child) != 0)
+		return -1;
+
+	/* Find the correct field in the structure */
+	die = &child;
+	do {
+		tag = dwarf_tag(die);
+		if (tag == DW_TAG_member) {
+			if (nfields == index)
+				break;
+			else
+				nfields++;
+		}
+	} while (!dwarf_siblingof(die, die));
+
+	if (nfields != index) {
+		ERRMSG("No member found at index %d.\n", index);
+		return -1;
+	}
+
+	/* Fill in the required info for the member */
+	if (!get_data_member_location(die, offset))
+		*offset = 0;
+
+	*name = dwarf_diename(die);
+	*m_die = dwarf_dieoffset(die);
+
+	get_die_type(die, &die_base);
+	if (dwarf_tag(&die_base) == DW_TAG_array_type) {
+		dwarf_info.array_length = 0;
+		get_data_array_length(die);
+		size = dwarf_info.array_length;
+	} else {
+		size = dwarf_bytesize(&die_base);
+	}
+
+	/* TODO
+	 * Correctly update fbits and nbits
+	 */
+	*nbits = *fbits = 0;
+
+	if (size < 0)
+		return 0;
+	else
+		return size;
+}
+
+/*
+ * Get the die attribute type
+ */
+int
+get_die_attr_type(unsigned long long die_off, int *type_flag,
+		unsigned long long *die_attr_off)
+{
+	Dwarf_Die result;
+
+	if (!die_attr_off)
+		return FALSE;
+
+	if (!get_die_from_offset((Dwarf_Off) die_off, &result)) {
+		ERRMSG("Can't find the DIE.\n");
+		return FALSE;
+	}
+
+	if (!get_die_type(&result, &result))
+		return FALSE;
+
+	*die_attr_off = dwarf_dieoffset(&result);
+	*type_flag = dwarf_tag(&result);
+	return TRUE;
+}
+
+/*
+ * Get name attribute given the die offset
+ */
+char *
+get_die_name(unsigned long long die_off)
+{
+	Dwarf_Die result;
+
+	if (!die_off)
+		return NULL;
+
+	if (!get_die_from_offset((Dwarf_Off) die_off, &result)) {
+		ERRMSG("Can't find the DIE.\n");
+		return NULL;
+	}
+
+	return dwarf_diename(&result);
+}
+
+/*
+ * Get length attribute given the die offset
+ */
+int
+get_die_length(unsigned long long die_off, int flag)
+{
+	Dwarf_Die result, die_base;
+
+	if (!die_off)
+		return FALSE;
+
+	if (!get_die_from_offset((Dwarf_Off) die_off, &result)) {
+		ERRMSG("Can't find the DIE.\n");
+		return FALSE;
+	}
+
+	if (flag)
+		return dwarf_bytesize(&result);
+
+	get_die_type(&result, &die_base);
+	if (dwarf_tag(&die_base) == DW_TAG_array_type) {
+		dwarf_info.array_length = 0;
+		get_data_array_length(&result);
+		return dwarf_info.array_length;
+	} else {
+		return dwarf_bytesize(&die_base);
+	}
+}
+
+/*
  * Set the dwarf_info with kernel/module debuginfo file information.
  */
 int
dwarf_info.h +7 -0
index 94d75ea..ce95aeb 100644
@@ -69,6 +69,13 @@ long get_array_length(char *name01, char *name02, unsigned int cmd);
 long get_enum_number(char *enum_name);
 int get_source_filename(char *structname, char *src_name, int cmd);
 long get_domain(char *symname, int cmd, unsigned long long *die);
+int get_die_nfields(unsigned long long die_off);
+int get_die_member(unsigned long long die_off, int index, long *offset,
+	char **name, int *nbits, int *fbits, unsigned long long *m_die);
+int get_die_attr_type(unsigned long long die_off, int *type_flag,
+	unsigned long long *die_attr_off);
+char *get_die_name(unsigned long long die_off);
+int get_die_length(unsigned long long die_off, int flag);
 int set_dwarf_debuginfo(char *mod_name, char *os_release, char *name_debuginfo, int fd_debuginfo);
 
 #endif  /* DWARF_INFO_H */
extension_eppic.c +175 -4

208 lines not on this page · raw message

extension_eppic.h +2 -0
index 100ef5b..a6ecf40 100644
@@ -29,6 +29,8 @@ int eppic_init(void);    /* Eppic initialize */
  * for makedumpfile extension.
  */
 
+#define MAX_ARRAY_DIMENSION 16
+
 /* member information */
 typedef MEMBER_S {
 

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

PATCH v2 6/7 Extend eppic built-in functions to include memset function

In reply to Aravinda Prasad

The memset function will be used to specify the virtual address and the length of the data to be scrubbed in the dump file from the eppic macro. makedumpfile will convert these requests into filter_info nodes which will be enqueued for filtering. Existing makedumpfile functionality reads the filter_info nodes and scrubs the data accordingly.

---
 erase_info.c      |   33 ++++++++++++++++++++++++++++++++-
 erase_info.h      |    1 +
 extension_eppic.c |   20 ++++++++++++++++++++
 extension_eppic.h |    2 ++
 4 files changed, 55 insertions(+), 1 deletions(-)
erase_info.c +32 -1
index 5d4cfd1..3eba12f 100644
@@ -66,6 +66,8 @@ struct filter_info {
 	int			erase_info_idx;	/* 0= invalid index */
 	int			size_idx;
 
+	int			erase_ch;
+
 	struct filter_info      *next;
 	unsigned short          nullify;
 };
@@ -1574,6 +1576,7 @@ update_filter_info(struct config_entry *filter_symbol,
 	fl_info->paddr   = vaddr_to_paddr(sym_addr);
 	fl_info->size    = size;
 	fl_info->nullify = filter_symbol->nullify;
+	fl_info->erase_ch = 'X';
 
 	if (insert_filter_info(fl_info)) {
 		fl_info->erase_info_idx = add_erase_info_node(filter_symbol);
@@ -1582,6 +1585,34 @@ update_filter_info(struct config_entry *filter_symbol,
 	return TRUE;
 }
 
+int
+update_filter_info_raw(unsigned long long sym_addr, int ch, int len)
+{
+	struct filter_info *fl_info;
+
+	fl_info = calloc(1, sizeof(struct filter_info));
+	if (fl_info == NULL) {
+		ERRMSG("Can't allocate filter info\n");
+		return FALSE;
+	}
+
+	fl_info->vaddr   = sym_addr;
+	fl_info->paddr   = vaddr_to_paddr(sym_addr);
+	fl_info->size    = len;
+	fl_info->nullify = 0;
+	fl_info->erase_ch = ch;
+
+	if (insert_filter_info(fl_info)) {
+		/* TODO
+		 * Add support to update erase information to the
+		 * resulting dump file
+		 */
+		fl_info->erase_info_idx = 0;
+		fl_info->size_idx = 0;
+	}
+	return TRUE;
+}
+
 static int
 initialize_iteration_entry(struct config_entry *ie,
 				char *type_name, unsigned char type_flag)
@@ -2014,7 +2045,7 @@ filter_data_buffer(unsigned char *buf, unsigned long long paddr,
 		if (fl_info.nullify)
 			memset(buf_ptr, 0, fl_info.size);
 		else
-			memset(buf_ptr, 'X', fl_info.size);
+			memset(buf_ptr, fl_info.erase_ch, fl_info.size);
 	}
 }
 
erase_info.h +1 -0
index 0e3c3a3..ec55599 100644
@@ -42,6 +42,7 @@ int gather_filter_info(void);
 void clear_filter_info(void);
 void filter_data_buffer(unsigned char *buf, unsigned long long paddr, size_t size);
 unsigned long get_size_eraseinfo(void);
+int update_filter_info_raw(unsigned long long, int, int);
 
 #endif /* _ERASE_INFO_H */
 
extension_eppic.c +20 -0
index f2540e1..037c1bf 100644
@@ -387,6 +387,22 @@ apiops icops = {
 	apifindsym
 };
 
+/* Extensions to built-in functions */
+VALUE_S *
+eppic_memset(VALUE_S *vaddr, VALUE_S *vch, VALUE_S *vlen)
+{
+	ull addr = eppic_getval(vaddr);
+	int len = eppic_getval(vlen);
+	int ch = eppic_getval(vch);
+
+	/*
+	 * Set the value at address from iaddr till iaddr + nbytes
+	 * to the value specified in variable ch
+	 */
+	update_filter_info_raw(addr, ch, len);
+	return eppic_makebtype(1);
+}
+
 
 /* Initialize eppic */
 int
@@ -400,6 +416,10 @@ eppic_init()
 		/* set the new function callback */
 		eppic_setcallback(reg_callback);
 
+		/* Extend built-in functions to include memset */
+		eppic_builtin("int memset(char *, int, int)",
+				(bf_t *)eppic_memset);
+
 		return 0;
 	}
 	return 1;
extension_eppic.h +2 -0
index a6ecf40..68ec273 100644
@@ -73,4 +73,6 @@ typedef TYPE_S {
 	ull rtype;  /* type_t a reference refers too */
 } type_t;
 
+extern int update_filter_info_raw(unsigned long long, int, int);
+
 #endif /* _EXTENSION_EPPIC_H */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw

PATCH v2 7/7 Support fully typed symbol access mode

In reply to Aravinda Prasad

Eppic enables access to symbols in two ways. The first, is a more natural mode in that it makes symbols available as fully typed entities. The second, is generic and treats all symbols as an address to data which then needs to be cast to the proper type explicitly. The former obviously enables an easier cut & pasting of target code into eppic code.

Currently generic symbol access mode is supported. This patch extends the functionality to include support for fully typed symbol access mode (which will be the default mode) in eppic macros. User can switch to generic symbol access mode by setting the following environmental variable - EPPIC_LEGACY_MODE. libeppic.a will take care of handling EPPIC_LEGACY_MODE. libeppic.a will pass NULL to VALUE_S* argument of apigetval() call back function if EPPIC_LEGACY_MODE is set.

Refer to http://code.google.com/p/eppic/ for more information.

---
 dwarf_info.c      |   52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |    2 ++
 extension_eppic.c |   24 ++++++++++++++++++++++++
 3 files changed, 78 insertions(+), 0 deletions(-)
dwarf_info.c +52 -0
index 7d07c41..84331f0 100644
@@ -119,6 +119,15 @@ is_search_domain(int cmd)
 }
 
 static int
+is_search_die(int cmd)
+{
+	if (cmd == DWARF_INFO_GET_DIE)
+		return TRUE;
+	else
+		return FALSE;
+}
+
+static int
 process_module (Dwfl_Module *dwflmod,
 		void **userdata __attribute__ ((unused)),
 		const char *name __attribute__ ((unused)),
@@ -844,6 +853,25 @@ search_domain(Dwarf_Die *die, int *found)
 }
 
 static void
+search_die(Dwarf_Die *die, int *found)
+{
+	const char *name;
+
+	do {
+		name = dwarf_diename(die);
+
+		if ((!name) || strcmp(name, dwarf_info.symbol_name))
+			continue;
+
+		if (found)
+			*found = TRUE;
+
+		dwarf_info.die_offset = dwarf_dieoffset(die);
+		return;
+	} while (!dwarf_siblingof(die, die));
+}
+
+static void
 search_die_tree(Dwarf_Die *die, int *found)
 {
 	Dwarf_Die child;
@@ -871,6 +899,9 @@ search_die_tree(Dwarf_Die *die, int *found)
 
 	else if (is_search_domain(dwarf_info.cmd))
 		search_domain(die, found);
+
+	else if (is_search_die(dwarf_info.cmd))
+		search_die(die, found);
 }
 
 static int
@@ -1431,6 +1462,27 @@ get_die_name(unsigned long long die_off)
 }
 
 /*
+ * Get the die offset given the die name
+ */
+unsigned long long
+get_die_offset(char *sysname)
+{
+	dwarf_info.cmd         = DWARF_INFO_GET_DIE;
+	dwarf_info.symbol_name = sysname;
+	dwarf_info.type_name   = NULL;
+	dwarf_info.struct_size = NOT_FOUND_STRUCTURE;
+	dwarf_info.die_offset  = 0;
+
+	if (!sysname)
+		return 0;
+
+	if (!get_debug_info())
+		return 0;
+
+	return (unsigned long long)dwarf_info.die_offset;
+}
+
+/*
  * Get length attribute given the die offset
  */
 int
dwarf_info.h +2 -0
index ce95aeb..7f02e87 100644
@@ -54,6 +54,7 @@ enum {
 	DWARF_INFO_GET_DOMAIN_REF,
 	DWARF_INFO_GET_DOMAIN_STRING,
 	DWARF_INFO_GET_DOMAIN_BASE,
+	DWARF_INFO_GET_DIE,
 };
 
 char *get_dwarf_module_name(void);
@@ -75,6 +76,7 @@ int get_die_member(unsigned long long die_off, int index, long *offset,
 int get_die_attr_type(unsigned long long die_off, int *type_flag,
 	unsigned long long *die_attr_off);
 char *get_die_name(unsigned long long die_off);
+unsigned long long get_die_offset(char *sysname);
 int get_die_length(unsigned long long die_off, int flag);
 int set_dwarf_debuginfo(char *mod_name, char *os_release, char *name_debuginfo, int fd_debuginfo);
 
extension_eppic.c +24 -0
index 037c1bf..331a7e6 100644
@@ -313,6 +313,30 @@ apigetval(char *name, ull *val, VALUE_S *value)
 		return 0;
 
 	*val = ptr;
+
+	if (!value)
+		return 1;
+
+	/* Support for fully typed symbol access */
+	ull type;
+	TYPE_S *stype;
+
+	type = get_die_offset(name);
+	stype = eppic_gettype(value);
+
+	apigetrtype(type, stype);
+
+	eppic_pushref(stype, 1);
+	eppic_setmemaddr(value, *val);
+	eppic_do_deref(1, value, value);
+
+	*val = eppic_getval(value);
+
+	if (!eppic_typeislocal(stype) && eppic_type_getidx(stype) > 100) {
+		char *tname = get_die_name(eppic_type_getidx(stype));
+		if (tname)
+			eppic_chktype(stype, tname);
+	}
 	return 1;
 }
 

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw
In reply to Aravinda Prasad

Hello Aravinda,

Quoted 21 lines · On Thu, 08 Nov 2012 19:07:52 +0530 Aravinda Prasad…

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Eppic is an interpreter that facilitates access to the symbol and type information stored in an executable image or a dump file. Eppic defines a language semantic which is similar to C. Eppic macros can be used to specify rules/commands to erase data in an image file. makedumpfile will interpret the rules/commands provided by eppic macros with the help of eppic library and will suitably erase the required data in a dump file. Eppic provides a lot of language constructs like conditional statements, logical and arithmetic operators, nested loops, functions, etc., to traverse nested lists and trees and conditionally erase data in the dump file, enabling users to literally erase any data in the dump file which is accessible through global symbols.

Quoted 6 lines

The series of patches integrates eppic with makdumpfile. These patches require eppic library libeppic.a and eppic_api.h header file. The libeppic.a library can be built from the eppic source code available at the following URL:

http://code.google.com/p/eppic/

Thank you for re-sending your patches, I'll merge them into v1.5.1-rc. And thanks for sending your test script, it's helpful for understanding how to use eppic macro.

Atsushi Kumagai

Quoted 35 lines

TODO:

- Currently, works only for symbols in vmlinux, extend it to module symbols
- Functionality support:
- Implement the following callback functions.
- apialignment
- apigetenum
- apigetdefs
- Other functionalities specified in the code with TODO tag
- Support specifying eppic macros in makedumpfile.conf file
- Update erase info

Changelog from v1 to v2:

- Re-based to v1.5.0
- Introduced EPPIC=on in makefile, and hence eppic is now optional
- Incorporated review comments from Atsushi
- Minor formatting changes

Regards,
Aravinda
---

Aravinda Prasad (7):
Initialize and setup eppic
makedumpfile and eppic interface layer
Eppic call back functions to query a dump image Implement apigetctype call back function Implement apimember and apigetrtype call back functions Extend eppic built-in functions to include memset function Support fully typed symbol access mode

Quoted 21 lines
 Makefile          |    7 +
 dwarf_info.c      |  367 +++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |   18 ++
 erase_info.c      |   83 +++++++++-
 erase_info.h      |    5 +
 extension_eppic.c |  451 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 extension_eppic.h |   78 +++++++++
 makedumpfile.c    |    7 +
 makedumpfile.h    |    6 +
 9 files changed, 1018 insertions(+), 4 deletions(-)
 create mode 100644 extension_eppic.c
 create mode 100644 extension_eppic.h
Signature · 7 lines
Aravinda Prasad


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw
In reply to Aravinda Prasad
Quoted 8 lines · Aravinda Prasad

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Hi Aravinda,

Had few questions.

- Which file will contain all the rules?

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

Given the fact that it does not reduce the size of core file significantly, I would think that it is better to post process vmcore to wipe out some symbols.

Thanks
Vivek

Quoted 54 lines

Eppic is an interpreter that facilitates access to the symbol and type information stored in an executable image or a dump file. Eppic defines a language semantic which is similar to C. Eppic macros can be used to specify rules/commands to erase data in an image file. makedumpfile will interpret the rules/commands provided by eppic macros with the help of eppic library and will suitably erase the required data in a dump file. Eppic provides a lot of language constructs like conditional statements, logical and arithmetic operators, nested loops, functions, etc., to traverse nested lists and trees and conditionally erase data in the dump file, enabling users to literally erase any data in the dump file which is accessible through global symbols.

The series of patches integrates eppic with makdumpfile. These patches require eppic library libeppic.a and eppic_api.h header file. The libeppic.a library can be built from the eppic source code available at the following URL:

http://code.google.com/p/eppic/

TODO:

- Currently, works only for symbols in vmlinux, extend it to module symbols
- Functionality support:
- Implement the following callback functions.
- apialignment
- apigetenum
- apigetdefs
- Other functionalities specified in the code with TODO tag
- Support specifying eppic macros in makedumpfile.conf file
- Update erase info

Changelog from v1 to v2:

- Re-based to v1.5.0
- Introduced EPPIC=on in makefile, and hence eppic is now optional
- Incorporated review comments from Atsushi
- Minor formatting changes

Regards,
Aravinda
---

Aravinda Prasad (7):
Initialize and setup eppic
makedumpfile and eppic interface layer
Eppic call back functions to query a dump image Implement apigetctype call back function Implement apimember and apigetrtype call back functions Extend eppic built-in functions to include memset function Support fully typed symbol access mode

Quoted 21 lines
 Makefile          |    7 +
 dwarf_info.c      |  367 +++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |   18 ++
 erase_info.c      |   83 +++++++++-
 erase_info.h      |    5 +
 extension_eppic.c |  451 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 extension_eppic.h |   78 +++++++++
 makedumpfile.c    |    7 +
 makedumpfile.h    |    6 +
 9 files changed, 1018 insertions(+), 4 deletions(-)
 create mode 100644 extension_eppic.c
 create mode 100644 extension_eppic.h
Signature · 7 lines
Aravinda Prasad


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw

Hi Vivek,

Quoted 10 lines · Vivek Goyal
Quoted 8 lines · Aravinda Prasad

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Quoted 5 lines

Hi Aravinda,

Had few questions.

- Which file will contain all the rules?

As of now rule files will not be provided by makedumpfile. However, writing a rule file is very easy - it is a C program.

Quoted 4 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

Quoted 3 lines

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

For the time being, it is only during post processing.

Quoted 4 lines

Given the fact that it does not reduce the size of core file significantly, I would think that it is better to post process vmcore to wipe out some symbols.

The main intention is to remove confidential information from the dump file, like ssh keys etc., which could be just few bytes, hence, may not reduce the size of the dump significantly.

Quoted 80 lines

Thanks
Vivek

Quoted 75 lines

Eppic is an interpreter that facilitates access to the symbol and type information stored in an executable image or a dump file. Eppic defines a language semantic which is similar to C. Eppic macros can be used to specify rules/commands to erase data in an image file. makedumpfile will interpret the rules/commands provided by eppic macros with the help of eppic library and will suitably erase the required data in a dump file. Eppic provides a lot of language constructs like conditional statements, logical and arithmetic operators, nested loops, functions, etc., to traverse nested lists and trees and conditionally erase data in the dump file, enabling users to literally erase any data in the dump file which is accessible through global symbols.

The series of patches integrates eppic with makdumpfile. These patches require eppic library libeppic.a and eppic_api.h header file. The libeppic.a library can be built from the eppic source code available at the following URL:

http://code.google.com/p/eppic/

TODO:

- Currently, works only for symbols in vmlinux, extend it to module symbols
- Functionality support:
- Implement the following callback functions.
- apialignment
- apigetenum
- apigetdefs
- Other functionalities specified in the code with TODO tag
- Support specifying eppic macros in makedumpfile.conf file
- Update erase info

Changelog from v1 to v2:

- Re-based to v1.5.0
- Introduced EPPIC=on in makefile, and hence eppic is now optional
- Incorporated review comments from Atsushi
- Minor formatting changes

Regards,
Aravinda
---

Aravinda Prasad (7):
Initialize and setup eppic
makedumpfile and eppic interface layer
Eppic call back functions to query a dump image Implement apigetctype call back function Implement apimember and apigetrtype call back functions Extend eppic built-in functions to include memset function Support fully typed symbol access mode

 Makefile          |    7 +
 dwarf_info.c      |  367 +++++++++++++++++++++++++++++++++++++++++++
 dwarf_info.h      |   18 ++
 erase_info.c      |   83 +++++++++-
 erase_info.h      |    5 +
 extension_eppic.c |  451 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 extension_eppic.h |   78 +++++++++
 makedumpfile.c    |    7 +
 makedumpfile.h    |    6 +
 9 files changed, 1018 insertions(+), 4 deletions(-)
 create mode 100644 extension_eppic.c
 create mode 100644 extension_eppic.h
Signature · 7 lines
Aravinda Prasad


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#13 lore raw
Quoted 16 lines · Aravinda Prasad

Hi Vivek,

Quoted 12 lines · Vivek Goyal
Quoted 8 lines · Aravinda Prasad

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Hi Aravinda,

Quoted 7 lines
Quoted 3 lines

Had few questions.

- Which file will contain all the rules?

As of now rule files will not be provided by makedumpfile. However, writing a rule file is very easy - it is a C program.

Can you give some details about how does it work and what's the correlation with makedumpfile.

Quoted 10 lines
Quoted 5 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

How would distributions handle it. Will we continue to build makedumpfile without EPPIC=on. Any increase in initramfs size increase is frowned upon in general.

Quoted 7 lines
Quoted 4 lines

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

For the time being, it is only during post processing.

Again, how distributions will handle it. If it is being integrated makedumpfile, as opposed to an stand alone utility, that means it makedumpfile needs to link against this library so that somebody can later filter out the symbols. And that means initramfs size bloat too?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#14 lore raw
Quoted 19 lines · Vivek Goyal
Quoted 18 lines · Aravinda Prasad

Hi Vivek,

Quoted 14 lines · Vivek Goyal
Quoted 8 lines · Aravinda Prasad

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Hi Aravinda,

Had few questions.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

- Which file will contain all the rules?

As of now rule files will not be provided by makedumpfile. However, writing a rule file is very easy - it is a C program.

Can you give some details about how does it work and what's the correlation with makedumpfile.

struct key in include/linux/key.h holds "authentication token"/"access credential"/"keyring". Suppose these entries should be scrubbed from the dumpfile. Then the keyring_name_hash hash table should be scanned and for each non-empty list, the entire list should be traversed and payload.value (or any other data) in struct key should be cleared.

Now the EPPIC macro looks like this:

int
key()

{
    int i;
    struct list_head *head;
    struct list_head *next, *prev;

    head = (struct list_head *)keyring_name_hash;

    for (i = 0; i < 32; i++)
    {
        next = (struct list_head *) head[i].next;
        prev = (struct list_head *) head[i].prev;

        if (!next)
            continue;

        do
        {
            struct key *mykey, *off = 0;

            mykey = (struct key *)((unsigned long)(next)
                      - ((unsigned long)&(off->type_data)));

            memset((char *)mykey->payload.value, 'X', 0x8);

            next = *(struct list_head **) mykey->type_data.link.next;
        } while (next != prev);
    }
    return 1;
}

The data can be cleared by specifying:
makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

makedumpfile with the help of eppic will interpret the macro key.c, traverses all the hash chains and erases paylod.value of struct key.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Quoted 7 lines
Quoted 3 lines

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

How would distributions handle it. Will we continue to build makedumpfile without EPPIC=on. Any increase in initramfs size increase is frowned upon in general.

We would like distributions to build makedumpfile with EPPIC=on.

I am not sure, but do you think ~1MB is too much increase to go with?

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

Quoted 6 lines
Quoted 1 line

For the time being, it is only during post processing.

Again, how distributions will handle it. If it is being integrated makedumpfile, as opposed to an stand alone utility, that means it makedumpfile needs to link against this library so that somebody can later filter out the symbols. And that means initramfs size bloat too?

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Quoted 9 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#15 lore raw

----- Original Message -----

Quoted 70 lines

struct key in include/linux/key.h holds "authentication token"/"access credential"/"keyring". Suppose these entries should be scrubbed from the dumpfile. Then the keyring_name_hash hash table should be scanned and for each non-empty list, the entire list should be traversed and payload.value (or any other data) in struct key should be cleared.

Now the EPPIC macro looks like this:

int
key()

{
    int i;
    struct list_head *head;
    struct list_head *next, *prev;

    head = (struct list_head *)keyring_name_hash;

    for (i = 0; i < 32; i++)
    {
        next = (struct list_head *) head[i].next;
        prev = (struct list_head *) head[i].prev;

        if (!next)
            continue;

        do
        {
            struct key *mykey, *off = 0;

            mykey = (struct key *)((unsigned long)(next)
                      - ((unsigned long)&(off->type_data)));

            memset((char *)mykey->payload.value, 'X', 0x8);

            next = *(struct list_head **) mykey->type_data.link.next;
        } while (next != prev);
    }
    return 1;
}

The data can be cleared by specifying:
makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

makedumpfile with the help of eppic will interpret the macro key.c, traverses all the hash chains and erases paylod.value of struct key.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Quoted 7 lines
Quoted 3 lines

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

How would distributions handle it. Will we continue to build makedumpfile without EPPIC=on. Any increase in initramfs size increase is frowned upon in general.

We would like distributions to build makedumpfile with EPPIC=on.

I am not sure, but do you think ~1MB is too much increase to go with?

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

For the time being, it is only during post processing.

Quoted 8 lines
Quoted 5 lines

Again, how distributions will handle it. If it is being integrated makedumpfile, as opposed to an stand alone utility, that means it makedumpfile needs to link against this library so that somebody can later filter out the symbols. And that means initramfs size bloat too?

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Just to clarify -- your example indicates that the vmlinux file is required for this facility to work, correct?

Quoted 1 line

makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

Clearly distros won't be putting the vmlinux file in the initramfs -- that's the whole reasoning behind vmcoreinfo. So the 99% of users that aren't interested in scrubbing will have to pay the penalty of the larger makedumpfile binary.

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#16 lore raw

On Thu, Nov 15, 2012 at 09:27:45AM -0500, Dave Anderson wrote:

[..]

Quoted 6 lines
Quoted 2 lines

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Just to clarify -- your example indicates that the vmlinux file is required for this facility to work, correct?

Quoted 6 lines
Quoted 1 line

makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

Clearly distros won't be putting the vmlinux file in the initramfs -- that's the whole reasoning behind vmcoreinfo. So the 99% of users that aren't interested in scrubbing will have to pay the penalty of the larger makedumpfile binary.

That's a good point Dave. We will never put debug compiled vmlinux in initramfs. Following two alternatives come to my mind.

- Either makedumpfile provides some kind of library to parse/read/write dump files and we can write another stand alone utility for scrubbing dump files (say, scrub-vmcore), and it can link against makedumpfile libraries to take advantage of existing code.

- Or, we just identify what we want to scrube and make that code part of makedumpfile. Export relevant data structures from kernel using vmcoreinfo.

I prefer to keep things simple and like second option better.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw
Quoted 11 lines · Vivek Goyal

On Thu, Nov 15, 2012 at 09:27:45AM -0500, Dave Anderson wrote:

[..]

Quoted 8 lines
Quoted 2 lines

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Just to clarify -- your example indicates that the vmlinux file is required for this facility to work, correct?

Quoted 1 line

makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

Quoted 7 lines
Quoted 4 lines

Clearly distros won't be putting the vmlinux file in the initramfs -- that's the whole reasoning behind vmcoreinfo. So the 99% of users that aren't interested in scrubbing will have to pay the penalty of the larger makedumpfile binary.

That's a good point Dave. We will never put debug compiled vmlinux in initramfs. Following two alternatives come to my mind.

As I mentioned, we don't need vmlinux in initramfs as filtering is done during post processing only.

Quoted 15 lines

- Either makedumpfile provides some kind of library to parse/read/write dump files and we can write another stand alone utility for scrubbing dump files (say, scrub-vmcore), and it can link against makedumpfile libraries to take advantage of existing code.

- Or, we just identify what we want to scrube and make that code part of makedumpfile. Export relevant data structures from kernel using vmcoreinfo.

I prefer to keep things simple and like second option better.

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#18 lore raw
Quoted 20 lines · Aravinda Prasad
Quoted 16 lines · Vivek Goyal

On Thu, Nov 15, 2012 at 09:27:45AM -0500, Dave Anderson wrote:

[..]

Quoted 12 lines
Quoted 2 lines

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Just to clarify -- your example indicates that the vmlinux file is required for this facility to work, correct?

Quoted 1 line

makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

Clearly distros won't be putting the vmlinux file in the initramfs -- that's the whole reasoning behind vmcoreinfo. So the 99% of users that aren't interested in scrubbing will have to pay the penalty of the larger makedumpfile binary.

Quoted 6 lines
Quoted 2 lines

That's a good point Dave. We will never put debug compiled vmlinux in initramfs. Following two alternatives come to my mind.

As I mentioned, we don't need vmlinux in initramfs as filtering is done during post processing only.

You are missing the point. The point is that despite the fact that scrubbing will never be done from initramfs, all the users will pay penalty for increased makedumpfile size.

So why not write a separate tool (scrub-vmcore) so that makedumpfile users don't pay the bloated size penatly.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#19 lore raw

Hello,

Quoted 28 lines · On Fri, 16 Nov 2012 09:36:49 -0500 Vivek Goyal <[email protected]> wrote:
Quoted 20 lines · Aravinda Prasad
Quoted 16 lines · Vivek Goyal

On Thu, Nov 15, 2012 at 09:27:45AM -0500, Dave Anderson wrote:

[..]

Quoted 12 lines
Quoted 2 lines

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Just to clarify -- your example indicates that the vmlinux file is required for this facility to work, correct?

Quoted 1 line

makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

Clearly distros won't be putting the vmlinux file in the initramfs -- that's the whole reasoning behind vmcoreinfo. So the 99% of users that aren't interested in scrubbing will have to pay the penalty of the larger makedumpfile binary.

Quoted 6 lines
Quoted 2 lines

That's a good point Dave. We will never put debug compiled vmlinux in initramfs. Following two alternatives come to my mind.

As I mentioned, we don't need vmlinux in initramfs as filtering is done during post processing only.

Quoted 6 lines

You are missing the point. The point is that despite the fact that scrubbing will never be done from initramfs, all the users will pay penalty for increased makedumpfile size.

So why not write a separate tool (scrub-vmcore) so that makedumpfile users don't pay the bloated size penatly.

As Vivek said, I think it's difficult to persuade almost all distributor to specify EPPIC=on.

So, it seems that separate tool is better way for both user and distributor. If you will make it, I will drop eppic support from v1.5.1 and export core functions from makedumpfile as library for such tools in the future.

Does this meet your purpose, Aravinda ?

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#20 lore raw

On 2012-11-20 15:17, Atsushi Kumagai wrote:

...

Quoted 12 lines
Quoted 11 lines
Quoted 4 lines

As I mentioned, we don't need vmlinux in initramfs as filtering is done during post processing only.

You are missing the point. The point is that despite the fact that scrubbing will never be done from initramfs, all the users will pay penalty for increased makedumpfile size.

So why not write a separate tool (scrub-vmcore) so that makedumpfile users don't pay the bloated size penatly.

Quoted 8 lines

As Vivek said, I think it's difficult to persuade almost all distributor to specify EPPIC=on.

So, it seems that separate tool is better way for both user and distributor. If you will make it, I will drop eppic support from v1.5.1 and export core functions from makedumpfile as library for such tools in the future.

Does this meet your purpose, Aravinda ?

Vivek, Atsushi,

How about distributions building two targets one with default makedumpfile and other with EPPIC=on, which can be done by a simple modification to Makefile. Default makedumpfile will go into initramfs and the one with EPPIC=on can be used for filtering. This will avoid initramfs bloat and will also save effort in developing a new tool and exporting makedumpfile functions to a library. Also if we build a new tool, eventually, we will end up sharing the existing makdumpfile code.

Quoted 10 lines

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#21 lore raw
Quoted 26 lines · Aravinda Prasad

On 2012-11-20 15:17, Atsushi Kumagai wrote:

...

Quoted 19 lines
Quoted 11 lines
Quoted 4 lines

As I mentioned, we don't need vmlinux in initramfs as filtering is done during post processing only.

You are missing the point. The point is that despite the fact that scrubbing will never be done from initramfs, all the users will pay penalty for increased makedumpfile size.

So why not write a separate tool (scrub-vmcore) so that makedumpfile users don't pay the bloated size penatly.

As Vivek said, I think it's difficult to persuade almost all distributor to specify EPPIC=on.

So, it seems that separate tool is better way for both user and distributor. If you will make it, I will drop eppic support from v1.5.1 and export core functions from makedumpfile as library for such tools in the future.

Quoted 8 lines
Quoted 1 line

Does this meet your purpose, Aravinda ?

Vivek, Atsushi,

How about distributions building two targets one with default makedumpfile and other with EPPIC=on, which can be done by a simple modification to Makefile.

That's not a good idea. Generally distributions don't ship multiple binaries with minor differences like that. I think going with a separate tool for doing this kind of filtering is better.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#22 lore raw
Quoted 32 lines · Vivek Goyal
Quoted 31 lines · Aravinda Prasad

On 2012-11-20 15:17, Atsushi Kumagai wrote:

...

Quoted 12 lines
Quoted 11 lines
Quoted 4 lines

As I mentioned, we don't need vmlinux in initramfs as filtering is done during post processing only.

You are missing the point. The point is that despite the fact that scrubbing will never be done from initramfs, all the users will pay penalty for increased makedumpfile size.

So why not write a separate tool (scrub-vmcore) so that makedumpfile users don't pay the bloated size penatly.

Quoted 8 lines

As Vivek said, I think it's difficult to persuade almost all distributor to specify EPPIC=on.

So, it seems that separate tool is better way for both user and distributor. If you will make it, I will drop eppic support from v1.5.1 and export core functions from makedumpfile as library for such tools in the future.

Does this meet your purpose, Aravinda ?

Vivek, Atsushi,

Quoted 7 lines
Quoted 3 lines

How about distributions building two targets one with default makedumpfile and other with EPPIC=on, which can be done by a simple modification to Makefile.

That's not a good idea. Generally distributions don't ship multiple binaries with minor differences like that. I think going with a separate tool for doing this kind of filtering is better.

Customers who want to filter their dump would certainly prefer a single tool and a single shot to throw away pages, scrub data and then compress, split or send the resulting dump to some target using ssh. This important functionality is lost if scrubbing is not part of makedumpfile and a separate tool is developed which just does scrubbing.

Quoted 9 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#23 lore raw

On Thu, Nov 22, 2012 at 10:44:49PM +0530, Aravinda Prasad wrote:

[..]

Quoted 5 lines

Customers who want to filter their dump would certainly prefer a single tool and a single shot to throw away pages, scrub data and then compress, split or send the resulting dump to some target using ssh. This important functionality is lost if scrubbing is not part of makedumpfile and a separate tool is developed which just does scrubbing.

Customers do filtering from initramfs context. It is not practical to save TB of memory and filter it later. And we just concluded that is it not practical to do scrubbing from initramfs due to need of vmlinux.

So doing filtering and doing scrubbing will not happen at the same time practically. So bloating size of makedumpfile does not make sense to me.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#24 lore raw
Quoted 9 lines · Vivek Goyal

On Thu, Nov 22, 2012 at 10:44:49PM +0530, Aravinda Prasad wrote:

[..]

Quoted 5 lines

Customers who want to filter their dump would certainly prefer a single tool and a single shot to throw away pages, scrub data and then compress, split or send the resulting dump to some target using ssh. This important functionality is lost if scrubbing is not part of makedumpfile and a separate tool is developed which just does scrubbing.

Quoted 8 lines

Customers do filtering from initramfs context. It is not practical to save TB of memory and filter it later. And we just concluded that is it not practical to do scrubbing from initramfs due to need of vmlinux.

So doing filtering and doing scrubbing will not happen at the same time practically. So bloating size of makedumpfile does not make sense to me.

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

Quoted 3 lines

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#25 lore raw
Quoted 23 lines · Aravinda Prasad
Quoted 9 lines · Vivek Goyal

On Thu, Nov 22, 2012 at 10:44:49PM +0530, Aravinda Prasad wrote:

[..]

Quoted 5 lines

Customers who want to filter their dump would certainly prefer a single tool and a single shot to throw away pages, scrub data and then compress, split or send the resulting dump to some target using ssh. This important functionality is lost if scrubbing is not part of makedumpfile and a separate tool is developed which just does scrubbing.

Quoted 8 lines

Customers do filtering from initramfs context. It is not practical to save TB of memory and filter it later. And we just concluded that is it not practical to do scrubbing from initramfs due to need of vmlinux.

So doing filtering and doing scrubbing will not happen at the same time practically. So bloating size of makedumpfile does not make sense to me.

Quoted 8 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

It will still show up in dynamic library dependencing using ldd. We will have to put some hack to exclude the leppic despite the fact that makedumpfile is dependent on it.

In F18, now we use dracut to build kdump initramfs. On command line we specify any extra binaries to be included and makedumpfile is one of those. Dracut will determine all the dependencies and automatically pull these in. So even if we don't use --eppic flag, dracult will pull in eppic shared library anyway.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#26 lore raw
Quoted 37 lines · Vivek Goyal
Quoted 23 lines · Aravinda Prasad
Quoted 9 lines · Vivek Goyal

On Thu, Nov 22, 2012 at 10:44:49PM +0530, Aravinda Prasad wrote:

[..]

Quoted 5 lines

Customers who want to filter their dump would certainly prefer a single tool and a single shot to throw away pages, scrub data and then compress, split or send the resulting dump to some target using ssh. This important functionality is lost if scrubbing is not part of makedumpfile and a separate tool is developed which just does scrubbing.

Quoted 8 lines

Customers do filtering from initramfs context. It is not practical to save TB of memory and filter it later. And we just concluded that is it not practical to do scrubbing from initramfs due to need of vmlinux.

So doing filtering and doing scrubbing will not happen at the same time practically. So bloating size of makedumpfile does not make sense to me.

Quoted 8 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

It will still show up in dynamic library dependencing using ldd. We will have to put some hack to exclude the leppic despite the fact that makedumpfile is dependent on it.

Quoted 5 lines

In F18, now we use dracut to build kdump initramfs. On command line we specify any extra binaries to be included and makedumpfile is one of those. Dracut will determine all the dependencies and automatically pull these in. So even if we don't use --eppic flag, dracult will pull in eppic shared library anyway.

I think ldd or dracut will not be able to make out that we are dependent on libeppic.so as we will be using the dlopen("libeppic.so, ...) call to dynamically load. No extra flags will be added to Makefile to specify -leppic while building makedumpfile. Hence, from my understanding, while building kdump initramfs, dracut cannot determine that we are dependent on eppic

Quoted 4 lines

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#27 lore raw

On Mon, Dec 03, 2012 at 08:05:54PM +0530, Aravinda Prasad wrote:

[..]

Quoted 20 lines
Quoted 13 lines
Quoted 8 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

It will still show up in dynamic library dependencing using ldd. We will have to put some hack to exclude the leppic despite the fact that makedumpfile is dependent on it.

Quoted 5 lines

In F18, now we use dracut to build kdump initramfs. On command line we specify any extra binaries to be included and makedumpfile is one of those. Dracut will determine all the dependencies and automatically pull these in. So even if we don't use --eppic flag, dracult will pull in eppic shared library anyway.

Quoted 6 lines

I think ldd or dracut will not be able to make out that we are dependent on libeppic.so as we will be using the dlopen("libeppic.so, ...) call to dynamically load. No extra flags will be added to Makefile to specify -leppic while building makedumpfile. Hence, from my understanding, while building kdump initramfs, dracut cannot determine that we are dependent on eppic

Ok, if dracut does not pull in libeppic and does not bloat size of initramfs, I am fine.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#28 lore raw

Hello Aravinda,

Quoted 30 lines · On Mon, 3 Dec 2012 13:40:01 -0500 Vivek Goyal <[email protected]> wrote:

On Mon, Dec 03, 2012 at 08:05:54PM +0530, Aravinda Prasad wrote:

[..]

Quoted 20 lines
Quoted 13 lines
Quoted 8 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

It will still show up in dynamic library dependencing using ldd. We will have to put some hack to exclude the leppic despite the fact that makedumpfile is dependent on it.

Quoted 5 lines

In F18, now we use dracut to build kdump initramfs. On command line we specify any extra binaries to be included and makedumpfile is one of those. Dracut will determine all the dependencies and automatically pull these in. So even if we don't use --eppic flag, dracult will pull in eppic shared library anyway.

Quoted 6 lines

I think ldd or dracut will not be able to make out that we are dependent on libeppic.so as we will be using the dlopen("libeppic.so, ...) call to dynamically load. No extra flags will be added to Makefile to specify -leppic while building makedumpfile. Hence, from my understanding, while building kdump initramfs, dracut cannot determine that we are dependent on eppic

Quoted 2 lines

Ok, if dracut does not pull in libeppic and does not bloat size of initramfs, I am fine.

I agree with this idea too.

However, the release date is closing in, so would you re-send the patch set based on v1.5.1 ? I will accept it as official feature in v1.5.2.

Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#29 lore raw
Quoted 35 lines · Atsushi Kumagai

Hello Aravinda,

Quoted 30 lines · On Mon, 3 Dec 2012 13:40:01 -0500 Vivek Goyal <[email protected]> wrote:

On Mon, Dec 03, 2012 at 08:05:54PM +0530, Aravinda Prasad wrote:

[..]

Quoted 26 lines
Quoted 13 lines
Quoted 8 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

It will still show up in dynamic library dependencing using ldd. We will have to put some hack to exclude the leppic despite the fact that makedumpfile is dependent on it.

Quoted 5 lines

In F18, now we use dracut to build kdump initramfs. On command line we specify any extra binaries to be included and makedumpfile is one of those. Dracut will determine all the dependencies and automatically pull these in. So even if we don't use --eppic flag, dracult will pull in eppic shared library anyway.

I think ldd or dracut will not be able to make out that we are dependent on libeppic.so as we will be using the dlopen("libeppic.so, ...) call to dynamically load. No extra flags will be added to Makefile to specify -leppic while building makedumpfile. Hence, from my understanding, while building kdump initramfs, dracut cannot determine that we are dependent on eppic

Quoted 8 lines
Quoted 2 lines

Ok, if dracut does not pull in libeppic and does not bloat size of initramfs, I am fine.

I agree with this idea too.

However, the release date is closing in, so would you re-send the patch set based on v1.5.1 ? I will accept it as official feature in v1.5.2.

sure. Thanks Atsushi

Quoted 4 lines

Thanks
Atsushi Kumagai

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#30 lore raw
In reply to Aravinda Prasad

----- Original Message -----

Quoted 10 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

OK, so based upon the most recent discussion here:

Re: [PATCH v2 0/7] makedumpfile security key filtering with eppic
http://lists.infradead.org/pipermail/kexec/2012-December/007452.html

Aravinda states that he will be posting an update for makedumpfile that will dlopen() a new "libeppic.so" shared object that is built with the libeppic.a library.

The upstream eppic git tree contains an "applications/crash" subdirectory that contains the code that creates a crash-utility-specific "eppic.so" file.

Accordingly, this new proposal will also require new source file(s) and a build process to create this new makedumpfile-specific "libeppic.so" shared object.

Arvinda -- will you be proposing an additional "applications/makedumpfile" subdirectory for the upstream eppic git tree that can be used to create the makedumpfile-specific libeppic.so file?

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#31 lore raw
Quoted 27 lines · Dave Anderson

----- Original Message -----

Quoted 10 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to makedumpfile code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building .so will be similar to what we have in crash.

OK, so based upon the most recent discussion here:

Re: [PATCH v2 0/7] makedumpfile security key filtering with eppic
http://lists.infradead.org/pipermail/kexec/2012-December/007452.html

Aravinda states that he will be posting an update for makedumpfile that will dlopen() a new "libeppic.so" shared object that is built with the libeppic.a library.

The upstream eppic git tree contains an "applications/crash" subdirectory that contains the code that creates a crash-utility-specific "eppic.so" file.

Quoted 7 lines

Accordingly, this new proposal will also require new source file(s) and a build process to create this new makedumpfile-specific "libeppic.so" shared object.

Arvinda -- will you be proposing an additional "applications/makedumpfile" subdirectory for the upstream eppic git tree that can be used to create the makedumpfile-specific libeppic.so file?

This is what I am planning.

A new extension_eppic.c file will be created under makedumpfile source directory. This file is equivalent to applications/crash/eppic.c in upstream eppic repository. A new target will be added to the Makefile of makedumpfile to build the shared library and to build this shared library libeppic.a would be required. The makedumpfile specific shared library will be named different - say eppic_mkdumpfile.so to avoid conflict with crash specific libeppic.so.

The reason for including extension_eppic.c under makedumpfile source is because it will be dependent on other functions in makedumpfile code like dwarf related calls etc. People modifying those functions should be aware of the callers in extension_eppic.c and if this is included in upstream eppic code, it will be easily overlooked (or may not be aware of its existence).

A flag --eppic will be added to makedumpfile which will dlopen the shared library. Few additional helper functions will be added to dwarf_info.c and erase_info.c - the same functions which are in v2 of the patchset.

Quoted 9 lines

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#32 lore raw
Quoted 10 lines

-----Original Message-----
From: Aravinda Prasad [mailto:[email protected]]
Sent: Friday, December 07, 2012 1:05 AM
To: Dave Anderson
Cc: [email protected]; [email protected]; [email protected]; Luc Chouinard; [email protected]; Atsushi Kumagai; Vivek Goyal; [email protected]
Subject: Re: [PATCH v2 0/7] makedumpfile security key filtering with

eppic

Quoted 12 lines · Dave Anderson
Quoted 8 lines

----- Original Message -----

Quoted 3 lines

Another approach is to dynamically load libeppic library - similar way how crash does it. No major changes will be done to

makedumpfile

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

code, except the addition of --eppic flag. Upon specifying --eppic flag makedumpfile will dlopen libeppic.so, which will have functionality to scrub the specified data. This will prevent makedumpfile bloat and will not affect the size of initramfs as --eppic is only specified during post processing. The distribution should build and ship libeppic.so and the procedure for building

.so

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

will be similar to what we have in crash.

OK, so based upon the most recent discussion here:

Re: [PATCH v2 0/7] makedumpfile security key filtering with eppic
http://lists.infradead.org/pipermail/kexec/2012-December/007452.html

Quoted 6 lines
Quoted 6 lines

Aravinda states that he will be posting an update for makedumpfile that will dlopen() a new "libeppic.so" shared object that is built with the libeppic.a library.

The upstream eppic git tree contains an "applications/crash" subdirectory that contains the code that creates a

crash-utility-specific

Quoted 4 lines

"eppic.so" file.

Quoted 3 lines

Accordingly, this new proposal will also require new source file(s) and a build process to create this new makedumpfile-specific

"libeppic.so"

Quoted 3 lines
Quoted 3 lines

shared object.

Arvinda -- will you be proposing an additional

"applications/makedumpfile"

Quoted 8 lines
Quoted 2 lines

subdirectory for the upstream eppic git tree that can be used to create the makedumpfile-specific libeppic.so file?

This is what I am planning.

A new extension_eppic.c file will be created under makedumpfile source directory. This file is equivalent to applications/crash/eppic.c in

upstream eppic

Quoted 1 line

repository. A new target will be added to the Makefile of makedumpfile

to build

Quoted 1 line

the shared library and to build this shared library libeppic.a would

be required.

Quoted 4 lines

The makedumpfile specific shared library will be named different - say eppic_mkdumpfile.so to avoid conflict with crash specific libeppic.so.

The reason for including extension_eppic.c under makedumpfile source

is

Quoted 1 line

because it will be dependent on other functions in makedumpfile code

like dwarf

Quoted 1 line

related calls etc. People modifying those functions should be aware of

the

Quoted 1 line

callers in extension_eppic.c and if this is included in upstream eppic

code, it will

Quoted 1 line

be easily overlooked (or may not be aware of its existence).

The same reasons exists for applications/crash/eppic.c. It only ended up on eppic's git for two reasons. To provide for a complete example of integration into a tool and, because I originally wrote it, I could support it from there without having to send a patch Dave's way.

I expect other applications to own their glue/integration file(s) and only grab libeppic from the git. These applications, will comply to libeppic's APIs and only fixes or new functions will be added to libeppic to support these applications and valid new requirements they may have. Like we did for mkdumpfile.

Same libeppic.so and same libeppic Makefile for all.

Quoted 2 lines

A flag --eppic will be added to makedumpfile which will dlopen the

shared

Quoted 1 line

library. Few additional helper functions will be added to dwarf_info.c

and

Quoted 14 lines

erase_info.c - the same functions which are in v2 of the patchset.

Quoted 9 lines

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Quoted 3 lines

--
Regards,
Aravinda

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#33 lore raw

On Fri, Dec 07, 2012 at 08:46:33AM -0500, Luc Chouinard wrote:

[..]

Quoted 27 lines
Quoted 4 lines

This is what I am planning.

A new extension_eppic.c file will be created under makedumpfile source directory. This file is equivalent to applications/crash/eppic.c in

upstream eppic

Quoted 1 line

repository. A new target will be added to the Makefile of makedumpfile

to build

Quoted 1 line

the shared library and to build this shared library libeppic.a would

be required.

Quoted 4 lines

The makedumpfile specific shared library will be named different - say eppic_mkdumpfile.so to avoid conflict with crash specific libeppic.so.

The reason for including extension_eppic.c under makedumpfile source

is

Quoted 1 line

because it will be dependent on other functions in makedumpfile code

like dwarf

Quoted 1 line

related calls etc. People modifying those functions should be aware of

the

Quoted 1 line

callers in extension_eppic.c and if this is included in upstream eppic

code, it will

Quoted 1 line

be easily overlooked (or may not be aware of its existence).

The same reasons exists for applications/crash/eppic.c. It only ended up on eppic's git for two reasons. To provide for a complete example of integration into a tool and, because I originally wrote it, I could support it from there without having to send a patch Dave's way.

Quoted 7 lines

I expect other applications to own their glue/integration file(s) and only grab libeppic from the git. These applications, will comply to libeppic's APIs and only fixes or new functions will be added to libeppic to support these applications and valid new requirements they may have. Like we did for mkdumpfile.

Same libeppic.so and same libeppic Makefile for all.

I am lost here. Can somebody explain this thing in layman's terms with simple example. (We have a function foo() in libeppic which makeudmpfile wants to call. Now what?)

Few questions come to my mind.

- What is glue logic and why do we need application specific glue logic to use this library.

- Assuming we need glue logic and assuming it is small enough, will it make sense to build statically build glue logic in application and build actual libeppic as shared object and do dlopen() on it.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#34 lore raw
Quoted 38 lines · Vivek Goyal

On Fri, Dec 07, 2012 at 08:46:33AM -0500, Luc Chouinard wrote:

[..]

Quoted 27 lines
Quoted 4 lines

This is what I am planning.

A new extension_eppic.c file will be created under makedumpfile source directory. This file is equivalent to applications/crash/eppic.c in

upstream eppic

Quoted 1 line

repository. A new target will be added to the Makefile of makedumpfile

to build

Quoted 1 line

the shared library and to build this shared library libeppic.a would

be required.

Quoted 4 lines

The makedumpfile specific shared library will be named different - say eppic_mkdumpfile.so to avoid conflict with crash specific libeppic.so.

The reason for including extension_eppic.c under makedumpfile source

is

Quoted 1 line

because it will be dependent on other functions in makedumpfile code

like dwarf

Quoted 1 line

related calls etc. People modifying those functions should be aware of

the

Quoted 1 line

callers in extension_eppic.c and if this is included in upstream eppic

code, it will

Quoted 1 line

be easily overlooked (or may not be aware of its existence).

The same reasons exists for applications/crash/eppic.c. It only ended up on eppic's git for two reasons. To provide for a complete example of integration into a tool and, because I originally wrote it, I could support it from there without having to send a patch Dave's way.

Quoted 7 lines

I expect other applications to own their glue/integration file(s) and only grab libeppic from the git. These applications, will comply to libeppic's APIs and only fixes or new functions will be added to libeppic to support these applications and valid new requirements they may have. Like we did for mkdumpfile.

Same libeppic.so and same libeppic Makefile for all.

Quoted 8 lines

I am lost here. Can somebody explain this thing in layman's terms with simple example. (We have a function foo() in libeppic which makeudmpfile wants to call. Now what?)

Few questions come to my mind.

- What is glue logic and why do we need application specific glue logic to use this library.

Consider this simple eppic macro, which prints the value of the global kernel variable "modules":

smod()

{
	struct list_head *mod;

	mod = (struct list_head *)modules;
	printf("Modules = %p \n", mod);
}

Eppic has logic to interpret this C program, but needs some help in resolving the actual value of "modules" (as well as members of struct list_head) and defines few call-back functions to help it. This generic logic forms libeppic.a. The glue logic implements these call back function to help libeppic.a.

Eg: whenever libeppic.a encounters struct list_head, it calls apimember("list_head") call back function to learn about the details of the members of the structure. Glue logic implement this call-back function and calls libeppic.a calls with information - eppic_member_sname("list_head", "next"), eppic_member_sname("list_head", "prev") etc., inside apimember("list_head").

In crash, the glue logic contains gdb related calls and in makedumpfile it contains dwarf related calls to get the structure members, value of global variables etc. However, libeppic.a does not care what calls we use in the background. It just needs the data. This makes it very generic and powerful.

Now the glue logic along with libeppic.a makes the shared library *.so (named eppic.so for crash, planning to name eppic_makedumpfile.so for makedumpfile), which applications can dlopen().

Hope this helps.

Quoted 5 lines

- Assuming we need glue logic and assuming it is small enough, will it make sense to build statically build glue logic in application and build actual libeppic as shared object and do dlopen() on it.

Glue logic is 450K lines of code (all of them in extension_eppic.c) and calls lot of eppic library calls, making this glue logic static requires libeppic.a to be statically included, which was our first approach.

Quoted 3 lines

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#35 lore raw
Quoted 89 lines · Aravinda Prasad
Quoted 38 lines · Vivek Goyal

On Fri, Dec 07, 2012 at 08:46:33AM -0500, Luc Chouinard wrote:

[..]

Quoted 27 lines
Quoted 4 lines

This is what I am planning.

A new extension_eppic.c file will be created under makedumpfile source directory. This file is equivalent to applications/crash/eppic.c in

upstream eppic

Quoted 1 line

repository. A new target will be added to the Makefile of makedumpfile

to build

Quoted 1 line

the shared library and to build this shared library libeppic.a would

be required.

Quoted 4 lines

The makedumpfile specific shared library will be named different - say eppic_mkdumpfile.so to avoid conflict with crash specific libeppic.so.

The reason for including extension_eppic.c under makedumpfile source

is

Quoted 1 line

because it will be dependent on other functions in makedumpfile code

like dwarf

Quoted 1 line

related calls etc. People modifying those functions should be aware of

the

Quoted 1 line

callers in extension_eppic.c and if this is included in upstream eppic

code, it will

Quoted 1 line

be easily overlooked (or may not be aware of its existence).

The same reasons exists for applications/crash/eppic.c. It only ended up on eppic's git for two reasons. To provide for a complete example of integration into a tool and, because I originally wrote it, I could support it from there without having to send a patch Dave's way.

Quoted 7 lines

I expect other applications to own their glue/integration file(s) and only grab libeppic from the git. These applications, will comply to libeppic's APIs and only fixes or new functions will be added to libeppic to support these applications and valid new requirements they may have. Like we did for mkdumpfile.

Same libeppic.so and same libeppic Makefile for all.

Quoted 8 lines

I am lost here. Can somebody explain this thing in layman's terms with simple example. (We have a function foo() in libeppic which makeudmpfile wants to call. Now what?)

Few questions come to my mind.

- What is glue logic and why do we need application specific glue logic to use this library.

Consider this simple eppic macro, which prints the value of the global kernel variable "modules":

smod()

{
	struct list_head *mod;

	mod = (struct list_head *)modules;
	printf("Modules = %p \n", mod);
}

Eppic has logic to interpret this C program, but needs some help in resolving the actual value of "modules" (as well as members of struct list_head) and defines few call-back functions to help it. This generic logic forms libeppic.a. The glue logic implements these call back function to help libeppic.a.

Eg: whenever libeppic.a encounters struct list_head, it calls apimember("list_head") call back function to learn about the details of the members of the structure. Glue logic implement this call-back function and calls libeppic.a calls with information - eppic_member_sname("list_head", "next"), eppic_member_sname("list_head", "prev") etc., inside apimember("list_head").

In crash, the glue logic contains gdb related calls and in makedumpfile it contains dwarf related calls to get the structure members, value of global variables etc. However, libeppic.a does not care what calls we use in the background. It just needs the data. This makes it very generic and powerful.

Now the glue logic along with libeppic.a makes the shared library *.so (named eppic.so for crash, planning to name eppic_makedumpfile.so for makedumpfile), which applications can dlopen().

Hope this helps.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

- Assuming we need glue logic and assuming it is small enough, will it make sense to build statically build glue logic in application and build actual libeppic as shared object and do dlopen() on it.

Glue logic is 450K lines of code (all of them in extension_eppic.c) and

Correction: it is just 450 lines of code not 450K !!

Quoted 9 lines

calls lot of eppic library calls, making this glue logic static requires libeppic.a to be statically included, which was our first approach.

Quoted 3 lines

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#36 lore raw
In reply to Dave Anderson

On 2012-11-15 19:57, Dave Anderson wrote:
...

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

Again, how distributions will handle it. If it is being integrated makedumpfile, as opposed to an stand alone utility, that means it makedumpfile needs to link against this library so that somebody can later filter out the symbols. And that means initramfs size bloat too?

Yes, makedumpfile needs to be linked against eppic library for filtering data and this will increase makedumpfile size and initramfs size too.

Just to clarify -- your example indicates that the vmlinux file is required for this facility to work, correct?

Yes vmlinux file is required

Quoted 7 lines
Quoted 2 lines

makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

Clearly distros won't be putting the vmlinux file in the initramfs -- that's the whole reasoning behind vmcoreinfo. So the 99% of users that aren't interested in scrubbing will have to pay the penalty of the larger makedumpfile binary.

vmlinux is not required in intitramfs as the filtering is done during post processing i.e., after the the dump ia saved

Quoted 3 lines

Dave

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#37 lore raw
In reply to Aravinda Prasad

On Thu, Nov 15, 2012 at 06:20:44PM +0530, Aravinda Prasad wrote:

[..]

Quoted 44 lines
Quoted 2 lines

Can you give some details about how does it work and what's the correlation with makedumpfile.

struct key in include/linux/key.h holds "authentication token"/"access credential"/"keyring". Suppose these entries should be scrubbed from the dumpfile. Then the keyring_name_hash hash table should be scanned and for each non-empty list, the entire list should be traversed and payload.value (or any other data) in struct key should be cleared.

Now the EPPIC macro looks like this:

int
key()

{
    int i;
    struct list_head *head;
    struct list_head *next, *prev;

    head = (struct list_head *)keyring_name_hash;

    for (i = 0; i < 32; i++)
    {
        next = (struct list_head *) head[i].next;
        prev = (struct list_head *) head[i].prev;

        if (!next)
            continue;

        do
        {
            struct key *mykey, *off = 0;

            mykey = (struct key *)((unsigned long)(next)
                      - ((unsigned long)&(off->type_data)));

            memset((char *)mykey->payload.value, 'X', 0x8);

            next = *(struct list_head **) mykey->type_data.link.next;
        } while (next != prev);
    }
    return 1;
}
Quoted 5 lines

The data can be cleared by specifying:
makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

makedumpfile with the help of eppic will interpret the macro key.c, traverses all the hash chains and erases paylod.value of struct key.

Ok, are these the only places where key is. Can a copy of it exist in some other buffers? We don't clear these.

Also, if key is the only issue, why not just write this logic in makedumpfile and provide another option, --clear-kernel-keys.

Why to introduce such generic scheme.

[..]

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

1MB bloat is significant given the fact that we reserve only 128MB for kdump kernel. Hence we need to review the benefits of this bloat very carefully.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#38 lore raw
Quoted 47 lines · Vivek Goyal

On Thu, Nov 15, 2012 at 06:20:44PM +0530, Aravinda Prasad wrote:

[..]

Quoted 44 lines
Quoted 2 lines

Can you give some details about how does it work and what's the correlation with makedumpfile.

struct key in include/linux/key.h holds "authentication token"/"access credential"/"keyring". Suppose these entries should be scrubbed from the dumpfile. Then the keyring_name_hash hash table should be scanned and for each non-empty list, the entire list should be traversed and payload.value (or any other data) in struct key should be cleared.

Now the EPPIC macro looks like this:

int
key()

{
    int i;
    struct list_head *head;
    struct list_head *next, *prev;

    head = (struct list_head *)keyring_name_hash;

    for (i = 0; i < 32; i++)
    {
        next = (struct list_head *) head[i].next;
        prev = (struct list_head *) head[i].prev;

        if (!next)
            continue;

        do
        {
            struct key *mykey, *off = 0;

            mykey = (struct key *)((unsigned long)(next)
                      - ((unsigned long)&(off->type_data)));

            memset((char *)mykey->payload.value, 'X', 0x8);

            next = *(struct list_head **) mykey->type_data.link.next;
        } while (next != prev);
    }
    return 1;
}
Quoted 8 lines
Quoted 5 lines

The data can be cleared by specifying:
makedumpfile -c -d 31 -x vmlinux --eppic key.c vmcore filtered_vmcore

makedumpfile with the help of eppic will interpret the macro key.c, traverses all the hash chains and erases paylod.value of struct key.

Ok, are these the only places where key is. Can a copy of it exist in some other buffers? We don't clear these.

I don't think a copy exist in other places

Quoted 5 lines

Also, if key is the only issue, why not just write this logic in makedumpfile and provide another option, --clear-kernel-keys.

Why to introduce such generic scheme.

key is not the only issue, it was just an example. There could be other things as well (data in socket buffers, device driver buffers, etc) which customers may consider sensitive/private and are interested in scrubbing.

Also this is an extension to the already existing generic solution implemented in makedumpfile, where rules can be specified using --config option. This extension is built on the existing infrastructure and provides a more flexible and powerful way to specify the data to be scrubbed. For eg, scrubbing the keyring data mentioned in one of my previous mails would not be possible with --config option.

Quoted 18 lines

[..]

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

1MB bloat is significant given the fact that we reserve only 128MB for kdump kernel. Hence we need to review the benefits of this bloat very carefully.

Thanks
Vivek

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#39 lore raw

On Fri, Nov 16, 2012 at 04:40:47PM +0530, Aravinda Prasad wrote:

[..]

Quoted 5 lines
Quoted 2 lines

Ok, are these the only places where key is. Can a copy of it exist in some other buffers? We don't clear these.

I don't think a copy exist in other places

I am wondering how does ssh work. User's private key is stored in .ssh/ and when authentication with server is happening then we must be signing something with that private key and most likely it will be in some buffer somewhere (user space buffer).

Quoted 13 lines
Quoted 6 lines

Also, if key is the only issue, why not just write this logic in makedumpfile and provide another option, --clear-kernel-keys.

Why to introduce such generic scheme.

key is not the only issue, it was just an example. There could be other things as well (data in socket buffers, device driver buffers, etc) which customers may consider sensitive/private and are interested in scrubbing.

Quoted 6 lines

Also this is an extension to the already existing generic solution implemented in makedumpfile, where rules can be specified using --config option. This extension is built on the existing infrastructure and provides a more flexible and powerful way to specify the data to be scrubbed. For eg, scrubbing the keyring data mentioned in one of my previous mails would not be possible with --config option.

I am not against building infrastructure to scrub vmcore. I am only concerned about size bloat of makedumpfile.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#40 lore raw
In reply to Aravinda Prasad

On Wed, Nov 14, 2012 at 10:36:53PM +0530, Aravinda Prasad wrote:

[..]

Quoted 3 lines

The main intention is to remove confidential information from the dump file, like ssh keys etc., which could be just few bytes, hence, may not reduce the size of the dump significantly.

Where are these ssh keys? Are these in user space memory or kernel memory? How would one know the location of these keys?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#41 lore raw
Quoted 9 lines · Vivek Goyal

On Wed, Nov 14, 2012 at 10:36:53PM +0530, Aravinda Prasad wrote:

[..]

Quoted 3 lines

The main intention is to remove confidential information from the dump file, like ssh keys etc., which could be just few bytes, hence, may not reduce the size of the dump significantly.

Where are these ssh keys? Are these in user space memory or kernel memory? How would one know the location of these keys?

Check out my previous mail which explains the keyring scenario in detail. ssh key was a bad example.

Quoted 9 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#42 lore raw
In reply to Aravinda Prasad

----- Original Message -----

Quoted 35 lines

Hi Vivek,

Quoted 10 lines · Vivek Goyal
Quoted 8 lines · Aravinda Prasad

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Quoted 5 lines

Hi Aravinda,

Had few questions.

- Which file will contain all the rules?

As of now rule files will not be provided by makedumpfile. However, writing a rule file is very easy - it is a C program.

Quoted 4 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

Quoted 1 line
Quoted 5 lines
Quoted 2 lines

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

For the time being, it is only during post processing.

By post-processing, I understand you to say that the system would be configured to do a full ELF vmcore dump, save it somewhere, and then somebody would do the post-processing at a later time?

Or is it possible to run makedumpfile again on a compressed kdump that was previously created at dump-time?

Quoted 9 lines
Quoted 4 lines

Given the fact that it does not reduce the size of core file significantly, I would think that it is better to post process vmcore to wipe out some symbols.

The main intention is to remove confidential information from the dump file, like ssh keys etc., which could be just few bytes, hence, may not reduce the size of the dump significantly.

So this would require you to first do a crash analysis on the unfiltered dumpfile, find out what you want to filter, write a script, and then run makedumpfile on the vmcore, correct?

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#43 lore raw
Quoted 42 lines · Dave Anderson

----- Original Message -----

Quoted 39 lines

Hi Vivek,

Quoted 10 lines · Vivek Goyal
Quoted 8 lines · Aravinda Prasad

makedumpfile security key filtering enhancement - Add Eppic language support (formerly known as SIAL) to specify rules to scrub data in a dumpfile. Eppic was previously part of crash source code repository.

The following series of patches enhance the makedumpfile to provide a more powerful way to specify rules and commands to traverse and erase complex data structures in a dump file by integrating Embeddable Pre-Processor and Interpreter for C (eppic).

Quoted 5 lines

Hi Aravinda,

Had few questions.

- Which file will contain all the rules?

As of now rule files will not be provided by makedumpfile. However, writing a rule file is very easy - it is a C program.

Quoted 4 lines

- What's the memory footprint of libeppic.a? Looks like this will be linked statically with makedumpfile, and how much is the size bloat of makedumpfile.

Memory footprint of libeppic.a is around 1MB. Yes, this will be statically linked to makedumpfile. Users should specify EPPIC=on while building the makedumpfile and hence linking libeppic.a is optional

Quoted 3 lines

- Is this supposed to work from kdump initramfs or it is supposed to be used on already saved dump (later during post processing).

Quoted 5 lines
Quoted 1 line

For the time being, it is only during post processing.

By post-processing, I understand you to say that the system would be configured to do a full ELF vmcore dump, save it somewhere, and then somebody would do the post-processing at a later time?

Yes exactly.

Quoted 3 lines

Or is it possible to run makedumpfile again on a compressed kdump that was previously created at dump-time?

Yes, it is possible to run makedumpfile with eppic again and again on compressed kdump, dump which excludes unnecessary pages and dumpfile which is already filtered by eppic.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Given the fact that it does not reduce the size of core file significantly, I would think that it is better to post process vmcore to wipe out some symbols.

Quoted 7 lines
Quoted 3 lines

The main intention is to remove confidential information from the dump file, like ssh keys etc., which could be just few bytes, hence, may not reduce the size of the dump significantly.

So this would require you to first do a crash analysis on the unfiltered dumpfile, find out what you want to filter, write a script, and then run makedumpfile on the vmcore, correct?

Need not be always. If users know what to erase in advance they can skip the crash analysis part. I had mentioned an example in my previous reply to Vivek on how to erase keyring data from struct key.

Quoted 4 lines

Dave

Signature · 8 lines
Regards,
Aravinda


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec