Skip to contentSkip to conversation

makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

PATCH In review Mahesh J Salgaonkar 5 participants 14 messages May 17, 2011 – Jul 15, 2011 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).

This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.

This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the makedumpfile.conf(8) man page.

The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.

I have tested these patches on x86_64 and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.

Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.

Please review the patchset and let me know your comments.

Thanks,
-Mahesh.
---

Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile

 IMPLEMENTATION      |    6 
 Makefile            |    8 
 diskdump_mod.h      |    2 
 makedumpfile.8      |   20 
 makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.conf   |  149 +++
 makedumpfile.conf.8 |  419 ++++++++
 makedumpfile.h      |  199 ++++
 8 files changed, 3394 insertions(+), 156 deletions(-)
 create mode 100644 makedumpfile.conf
 create mode 100644 makedumpfile.conf.8
Signature · 6 lines
Signature

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

Hi Mahesh,

Thank you for the patches.

But now I am out of the makedumpfile devel and other OSS devels.
The main maintainer of makedumpfile is Tachibana-san ([email protected]) and he will help you.

Thanks
Ken'ichi Ohmichi

--

Quoted 50 lines · On Wed, 18 May 2011 01:29:06 +0530 Mahesh J Salgaonkar…

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).

This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.

This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the makedumpfile.conf(8) man page.

The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.

I have tested these patches on x86_64 and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.

Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.

Please review the patchset and let me know your comments.

Thanks,
-Mahesh.
---

Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile

Quoted 14 lines
 IMPLEMENTATION      |    6 
 Makefile            |    8 
 diskdump_mod.h      |    2 
 makedumpfile.8      |   20 
 makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.conf   |  149 +++
 makedumpfile.conf.8 |  419 ++++++++
 makedumpfile.h      |  199 ++++
 8 files changed, 3394 insertions(+), 156 deletions(-)
 create mode 100644 makedumpfile.conf
 create mode 100644 makedumpfile.conf.8
Signature · 1 line
Signature

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw
In reply to Mahesh J Salgaonkar
Quoted 10 lines · Mahesh J Salgaonkar

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).

This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.

Mahesh,

Apart from security keys, what are the other examples of data which needs to be removed?

By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.

Thanks
Vivek

Quoted 39 lines

This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the makedumpfile.conf(8) man page.

The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.

I have tested these patches on x86_64 and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.

Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.

Please review the patchset and let me know your comments.

Thanks,
-Mahesh.
---

Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile

Quoted 19 lines
 IMPLEMENTATION      |    6 
 Makefile            |    8 
 diskdump_mod.h      |    2 
 makedumpfile.8      |   20 
 makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.conf   |  149 +++
 makedumpfile.conf.8 |  419 ++++++++
 makedumpfile.h      |  199 ++++
 8 files changed, 3394 insertions(+), 156 deletions(-)
 create mode 100644 makedumpfile.conf
 create mode 100644 makedumpfile.conf.8
Signature · 6 lines
Signature

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

----- Original Message -----

Quoted 17 lines · Mahesh J Salgaonkar
Quoted 10 lines

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).

This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.

Mahesh,

Apart from security keys, what are the other examples of data which needs to be removed?

Quoted 8 lines

By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.

Thanks
Vivek

As I said when the first patch-set was proposed, there is no way that I am going to put code into the crash utility that will somehow "work around" memory that has been poisoned. The only thing I plan to do is to check whether the new segments exist in the compressed kdump or ELF kdump headers -- and if they do, print a warning message during invocation informing the user of that fact. If Mahesh wants to dump the contents of the sections, he can post a patch to the crash-utility list.

And then, whatever breaks, breaks -- it is not the crash utility's problem to handle. Just looking at the examples in the documentation made me nervous -- IIRC the kernel "vmlist" pointer, and "vm_struct.addr" are shown -- both of which would cripple the crash utility.

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

A message that is not in the archive (sent off-list or not yet received): OF341326C7.0D81F89A-ONC125789B.002DB6ED-C125789B.002FC7A0@de.ibm.com

#5 lore raw
Quoted 11 lines · Reinhard Buendgen

Hi,

to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload within) of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their data will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions of that information sitting in some buffers to a service organization.

So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.

So atleast for above examples it does not sound as if we need symbol erase infrastructure.

Quoted 5 lines

to answer Daves concern: there is no intention that crash should ever look into the erased structures. In theroy it should not be needed because the contents of structures to be deleted should be irrelevant to kernel debugging.

So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?

I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

Vivek,

I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.

So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).

Allowing to configure makedumpfile allows each distribution and each platform to provide appropriate filters.

Mit freundlichen Grüßen/Best Regards/Cordialement

Reinhard

Dr. Reinhard Bündgen
RAS & Crypto Architect for Linux on System z
Virtualization and Systems Management

Mail:[email protected]
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456

IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

From:   Vivek Goyal <[email protected]>
To:     Reinhard Buendgen/Germany/IBM@IBMDE

Cc: Dave Anderson <[email protected]>, Ananth N Mavinakayanahalli
<[email protected]>, [email protected], Mahesh J Salgaonkar <[email protected]>, "Ken'ichi Ohmichi" <[email protected]>, V Srivatsa <[email protected]>

Date:   25.05.2011 21:53
Subject:        Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement 

to filter out kernel data from vmcore

Quoted 5 lines · Reinhard Buendgen

Hi,

to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload

within)

Quoted 3 lines

of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their

data

Quoted 4 lines

will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions

of that information sitting in some buffers to a service organization.

So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.

So atleast for above examples it does not sound as if we need symbol erase infrastructure.

Quoted 2 lines

to answer Daves concern: there is no intention that crash should ever

look

Quoted 1 line

into the erased structures. In theroy it should not be needed because

the

Quoted 2 lines

contents of structures to be deleted should be irrelevant to kernel debugging.

So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?

I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw
Quoted 10 lines · Reinhard Buendgen

Vivek,

I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.

So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).

Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.

When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.

Quoted 4 lines

Allowing to configure makedumpfile allows each distribution and each platform to provide appropriate filters.

I was thinking that there are very few symbols and these will not change frequently. makdumpfile is already dependent on some kenrel data strcutres and symbols for doing filtering.

But looks like you are planning to filter out lots of symbols. So in that case agreed that hardcoding is not a good idea.

Thanks
Vivek

Quoted 69 lines

Mit freundlichen Grüßen/Best Regards/Cordialement

Reinhard

Dr. Reinhard Bündgen
RAS & Crypto Architect for Linux on System z
Virtualization and Systems Management

Mail:[email protected]
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456

IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

From:   Vivek Goyal <[email protected]>
To:     Reinhard Buendgen/Germany/IBM@IBMDE

Cc: Dave Anderson <[email protected]>, Ananth N Mavinakayanahalli
<[email protected]>, [email protected], Mahesh J Salgaonkar <[email protected]>, "Ken'ichi Ohmichi" <[email protected]>, V Srivatsa <[email protected]>

Date:   25.05.2011 21:53
Subject:        Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement 

to filter out kernel data from vmcore

Quoted 5 lines · Reinhard Buendgen

Hi,

to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload

within)

Quoted 3 lines

of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their

data

Quoted 4 lines

will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions

of that information sitting in some buffers to a service organization.

So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.

So atleast for above examples it does not sound as if we need symbol erase infrastructure.

Quoted 2 lines

to answer Daves concern: there is no intention that crash should ever

look

Quoted 1 line

into the erased structures. In theroy it should not be needed because

the

Quoted 2 lines

contents of structures to be deleted should be irrelevant to kernel debugging.

So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?

Quoted 9 lines

I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw
Quoted 12 lines · Vivek Goyal
Quoted 10 lines · Reinhard Buendgen

Vivek,

I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.

So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).

Quoted 7 lines

Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.

When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.

The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).

Quoted 98 lines
Quoted 4 lines

Allowing to configure makedumpfile allows each distribution and each platform to provide appropriate filters.

I was thinking that there are very few symbols and these will not change frequently. makdumpfile is already dependent on some kenrel data strcutres and symbols for doing filtering.

But looks like you are planning to filter out lots of symbols. So in that case agreed that hardcoding is not a good idea.

Thanks
Vivek

Quoted 69 lines

Mit freundlichen Grüßen/Best Regards/Cordialement

Reinhard

Dr. Reinhard Bündgen
RAS & Crypto Architect for Linux on System z
Virtualization and Systems Management

Mail:[email protected]
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456

IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

From:   Vivek Goyal <[email protected]>
To:     Reinhard Buendgen/Germany/IBM@IBMDE

Cc: Dave Anderson <[email protected]>, Ananth N Mavinakayanahalli
<[email protected]>, [email protected], Mahesh J Salgaonkar <[email protected]>, "Ken'ichi Ohmichi" <[email protected]>, V Srivatsa <[email protected]>

Date:   25.05.2011 21:53
Subject:        Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement 

to filter out kernel data from vmcore

Quoted 5 lines · Reinhard Buendgen

Hi,

to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload

within)

Quoted 3 lines

of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their

data

Quoted 4 lines

will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions

of that information sitting in some buffers to a service organization.

So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.

So atleast for above examples it does not sound as if we need symbol erase infrastructure.

Quoted 2 lines

to answer Daves concern: there is no intention that crash should ever

look

Quoted 1 line

into the erased structures. In theroy it should not be needed because

the

Quoted 2 lines

contents of structures to be deleted should be irrelevant to kernel debugging.

So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?

Quoted 9 lines

I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 6 lines
Mahesh J Salgaonkar

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#9 lore raw

Hi Mahesh,

(I'm back to makedumpfile devel for merging this patchset,
because Tachibana-san is busy.)

Sorry for replying an old mail.

Quoted 16 lines · On Fri, 27 May 2011 12:26:21 +0530 Mahesh J Salgaonkar…
Quoted 15 lines · Vivek Goyal
Quoted 10 lines · Reinhard Buendgen

Vivek,

I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.

So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).

Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.

Quoted 8 lines
Quoted 4 lines

When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.

The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).

IIUC, there are be 2 dumpfiles on customer site by the above approach. The one is with privacy/secret data, and another is without.

If correct, I like this approach because a customer can have two options when the crash utility cannot read a dumpfile without privacy/secret data on support site.

First option:
For digging a problem, a customer sends a dumpfile with privacy/secret data to support site.

Second option:
For protecting privacy/secret data, a customer gives up digging a problem.

Thanks
Ken'ichi Ohmichi

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw

Hi Ken'ichi,

Quoted 5 lines · Ken'ichi Ohmichi

Hi Mahesh,

(I'm back to makedumpfile devel for merging this patchset,
because Tachibana-san is busy.)

Good to see that you are back.

Quoted 26 lines

Sorry for replying an old mail.

Quoted 21 lines · On Fri, 27 May 2011 12:26:21 +0530 Mahesh J Salgaonkar…
Quoted 19 lines · Vivek Goyal
Quoted 10 lines · Reinhard Buendgen

Vivek,

I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.

So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).

Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.

When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.

Quoted 6 lines
Quoted 3 lines

The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).

IIUC, there are be 2 dumpfiles on customer site by the above approach. The one is with privacy/secret data, and another is without.

Correct.

Quoted 4 lines

If correct, I like this approach because a customer can have two options when the crash utility cannot read a dumpfile without privacy/secret data on support site.

The crash utility would just display a warning message early on during invocation. Most of the time crash tool will be able to read/analyze the dump unless someone scrubs out the data on which crash utility is dependant on. And as I mentioned previously, this is intended just as a security filter and not to be used as detrimental to crash's analysis of the dump.

Quoted 11 lines

First option:
For digging a problem, a customer sends a dumpfile with privacy/secret data to support site.

Second option:
For protecting privacy/secret data, a customer gives up digging a problem.

Thanks
Ken'ichi Ohmichi

Thanks,
-Mahesh.

Quoted 5 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 6 lines
Mahesh J Salgaonkar

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#11 lore raw

Hi Mahesh,

Thank you for quick response.

Quoted 15 lines · On Fri, 15 Jul 2011 12:40:29 +0530 Mahesh J Salgaonkar…
Quoted 7 lines
Quoted 4 lines

The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).

IIUC, there are be 2 dumpfiles on customer site by the above approach. The one is with privacy/secret data, and another is without.

Correct.

Quoted 4 lines

If correct, I like this approach because a customer can have two options when the crash utility cannot read a dumpfile without privacy/secret data on support site.

Quoted 5 lines

The crash utility would just display a warning message early on during invocation. Most of the time crash tool will be able to read/analyze the dump unless someone scrubs out the data on which crash utility is dependant on. And as I mentioned previously, this is intended just as a security filter and not to be used as detrimental to crash's analysis of the dump.

I see, this feature will not be a problem in most cases. However a customer can create unreadable dumpfile by using this feature, and current makedumpfile does not have such feature.

Of course unreadable dumpfile is due to wrong makedumpfile.conf, and I like this feature anyway because a customer has options.

Thanks
Ken'ichi Ohmichi

Quoted 6 lines
Quoted 6 lines

First option:
For digging a problem, a customer sends a dumpfile with privacy/secret data to support site.

Second option:
For protecting privacy/secret data, a customer gives up digging a problem.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw
In reply to Vivek Goyal
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Mahesh J Salgaonkar

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).

Quoted 8 lines
Quoted 3 lines

This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.

Mahesh,

Apart from security keys, what are the other examples of data which needs to be removed?

The idea is to allow customer to remove any such confidential data that he thinks should not be disclosed outside. The framework is generic enough to accommodate any new such fields. The is intended just as a security filter and not to be used as detrimental to crash's analysis of the dump.

Quoted 6 lines

By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.

The data to be erased is poisoned with character 'X' (58 in Hex).

The last two patches 7/8 and 8/8 introduces eraseinfo section into filtered compressed kdump and ELF kdump file. The compressed kdump file now carries additional fields namely offset_eraseinfo and size_eraseinfo in kdump sub header that can help crash to identify whether filtering is been done. Similarly, ELF kdump file now contains new ELF note of type NT_ERASEINFO, that can help crash to identify whether the filtering has been done or not.

Thanks,
-Mahesh.

Quoted 62 lines

Thanks
Vivek

Quoted 58 lines

This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the makedumpfile.conf(8) man page.

The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.

I have tested these patches on x86_64 and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.

Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.

Please review the patchset and let me know your comments.

Thanks,
-Mahesh.
---

Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile

 IMPLEMENTATION      |    6 
 Makefile            |    8 
 diskdump_mod.h      |    2 
 makedumpfile.8      |   20 
 makedumpfile.c      | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++---
 makedumpfile.conf   |  149 +++
 makedumpfile.conf.8 |  419 ++++++++
 makedumpfile.h      |  199 ++++
 8 files changed, 3394 insertions(+), 156 deletions(-)
 create mode 100644 makedumpfile.conf
 create mode 100644 makedumpfile.conf.8
Signature · 6 lines
Signature

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Signature · 6 lines
Mahesh J Salgaonkar

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#13 lore raw
Quoted 13 lines · Mahesh J Salgaonkar
Quoted 12 lines · Vivek Goyal
Quoted 10 lines · Mahesh J Salgaonkar

Hi All,

Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).

This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.

Quoted 8 lines
Quoted 4 lines

Mahesh,

Apart from security keys, what are the other examples of data which needs to be removed?

The idea is to allow customer to remove any such confidential data that he thinks should not be disclosed outside. The framework is generic enough to accommodate any new such fields.

So what is that info which customer whould like to erase (apart from security keys?) Do we have some more concrete examples?

Quoted 3 lines

The is intended just as a
security filter and not to be used as detrimental to crash's analysis of the dump.

How would we know in advance that what symbols crash is dependent on to even open crash dump and we are not erasing one of those symbols. In other mail Dave mentioned two symbols (vmlist and one more) which crash is dependent on and we seem to be clearing that.

I am worried about filtering out symbols which make crash non functional. Dave mentioned that one can remove the pointer which contains the list of modules etc. I am worried what if module caused the crash and we break the chain in such a way that reaching to module data structures is not possible anymore.

If security key is only example, should we just hardcode that in makedumpfile to wipe that info instead of creating a generic framework for this. If we give user too much flexibility they might delete way too many symbols to be safe and that might result in un-analyzable crash dump.

Quoted 10 lines
Quoted 7 lines

By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.

The data to be erased is poisoned with character 'X' (58 in Hex).

Quoted 7 lines

The last two patches 7/8 and 8/8 introduces eraseinfo section into filtered compressed kdump and ELF kdump file. The compressed kdump file now carries additional fields namely offset_eraseinfo and size_eraseinfo in kdump sub header that can help crash to identify whether filtering is been done. Similarly, ELF kdump file now contains new ELF note of type NT_ERASEINFO, that can help crash to identify whether the filtering has been done or not.

And how this information is useful to crash? I mean why would crash care whether filtering has been done or not and if filtering has been done then do what?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#14 lore raw

----- Original Message -----

Quoted 11 lines
Quoted 10 lines

The data to be erased is poisoned with character 'X' (58 in Hex).

The last two patches 7/8 and 8/8 introduces eraseinfo section into filtered compressed kdump and ELF kdump file. The compressed kdump file now carries additional fields namely offset_eraseinfo and size_eraseinfo in kdump sub header that can help crash to identify whether filtering is been done. Similarly, ELF kdump file now contains new ELF note of type NT_ERASEINFO, that can help crash to identify whether the filtering has been done or not.

Quoted 3 lines

And how this information is useful to crash? I mean why would crash care whether filtering has been done or not and if filtering has been done then do what?

It's not particularly useful to crash per se -- but rather it would be useful to a user trying to analyze the dumpfile. If a prominent warning message is displayed early on during invocation, then it won't be a big surprise if/when things are subsequently broken/crippled.

Dave

Quoted 2 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec