makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore
Subsystems (2)
v2 cover makedumpfile: makedumpfile enhancement to filter out kernel data from vmcore
Hi All,
Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).
This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.
This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the makedumpfile.conf(8) man page.
The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.
I have tested these patches on x86_64 and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.
Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.
Please review the patchset and let me know your comments.
Thanks,
-Mahesh.
---
Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile
IMPLEMENTATION | 6 Makefile | 8 diskdump_mod.h | 2 makedumpfile.8 | 20 makedumpfile.c | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++--- makedumpfile.conf | 149 +++ makedumpfile.conf.8 | 419 ++++++++ makedumpfile.h | 199 ++++ 8 files changed, 3394 insertions(+), 156 deletions(-) create mode 100644 makedumpfile.conf create mode 100644 makedumpfile.conf.8
Signature · 6 lines
Signature _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hi Mahesh,
Thank you for the patches.
But now I am out of the makedumpfile devel and other OSS devels.
The main maintainer of makedumpfile is Tachibana-san ([email protected]) and he will help you.
Thanks
Ken'ichi Ohmichi
--
Quoted 50 lines · On Wed, 18 May 2011 01:29:06 +0530 Mahesh J Salgaonkar…
Hi All,
Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).
This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.
This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the
makedumpfile.conf(8)man page.The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.
I have tested these patches on
x86_64and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.Please review the patchset and let me know your comments.
Thanks,
-Mahesh.
---Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile
Quoted 14 lines
IMPLEMENTATION | 6 Makefile | 8 diskdump_mod.h | 2 makedumpfile.8 | 20 makedumpfile.c | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++--- makedumpfile.conf | 149 +++ makedumpfile.conf.8 | 419 ++++++++ makedumpfile.h | 199 ++++ 8 files changed, 3394 insertions(+), 156 deletions(-) create mode 100644 makedumpfile.conf create mode 100644 makedumpfile.conf.8Signature · 1 line
Signature
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · Mahesh J Salgaonkar
Hi All,
Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).
This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.
Mahesh,
Apart from security keys, what are the other examples of data which needs to be removed?
By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.
Thanks
Vivek
Quoted 39 lines
This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the
makedumpfile.conf(8)man page.The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.
I have tested these patches on
x86_64and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.Please review the patchset and let me know your comments.
Thanks,
-Mahesh.
---Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfile
Quoted 19 lines
IMPLEMENTATION | 6 Makefile | 8 diskdump_mod.h | 2 makedumpfile.8 | 20 makedumpfile.c | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++--- makedumpfile.conf | 149 +++ makedumpfile.conf.8 | 419 ++++++++ makedumpfile.h | 199 ++++ 8 files changed, 3394 insertions(+), 156 deletions(-) create mode 100644 makedumpfile.conf create mode 100644 makedumpfile.conf.8Signature · 6 lines
Signature _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
----- Original Message -----
Quoted 17 lines · Mahesh J Salgaonkar
Quoted 10 lines
Hi All,
Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).
This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.
Mahesh,
Apart from security keys, what are the other examples of data which needs to be removed?
Quoted 8 lines
By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.
Thanks
Vivek
As I said when the first patch-set was proposed, there is no way that I am going to put code into the crash utility that will somehow "work around" memory that has been poisoned. The only thing I plan to do is to check whether the new segments exist in the compressed kdump or ELF kdump headers -- and if they do, print a warning message during invocation informing the user of that fact. If Mahesh wants to dump the contents of the sections, he can post a patch to the crash-utility list.
And then, whatever breaks, breaks -- it is not the crash utility's problem to handle. Just looking at the examples in the documentation made me nervous -- IIRC the kernel "vmlist" pointer, and "vm_struct.addr" are shown -- both of which would cripple the crash utility.
Dave
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
A message that is not in the archive (sent off-list or not yet received): OF341326C7.0D81F89A-ONC125789B.002DB6ED-C125789B.002FC7A0@de.ibm.com
Quoted 11 lines · Reinhard Buendgen
Hi,
to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload within) of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their data will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions of that information sitting in some buffers to a service organization.
So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.
So atleast for above examples it does not sound as if we need symbol erase infrastructure.
Quoted 5 lines
to answer Daves concern: there is no intention that crash should ever look into the erased structures. In theroy it should not be needed because the contents of structures to be deleted should be irrelevant to kernel debugging.
So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?
I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Vivek,
I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.
So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).
Allowing to configure makedumpfile allows each distribution and each platform to provide appropriate filters.
Mit freundlichen Grüßen/Best Regards/Cordialement
Reinhard
Dr. Reinhard Bündgen
RAS & Crypto Architect for Linux on System z
Virtualization and Systems Management
Mail:[email protected]
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456
IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
From: Vivek Goyal <[email protected]> To: Reinhard Buendgen/Germany/IBM@IBMDE
Cc: Dave Anderson <[email protected]>, Ananth N Mavinakayanahalli
<[email protected]>, [email protected], Mahesh J Salgaonkar <[email protected]>, "Ken'ichi Ohmichi" <[email protected]>, V Srivatsa <[email protected]>
Date: 25.05.2011 21:53 Subject: Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancement
to filter out kernel data from vmcore
Quoted 5 lines · Reinhard Buendgen
Hi,
to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload
within)
Quoted 3 lines
of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their
data
Quoted 4 lines
will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions
of that information sitting in some buffers to a service organization.
So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.
So atleast for above examples it does not sound as if we need symbol erase infrastructure.
Quoted 2 lines
to answer Daves concern: there is no intention that crash should ever
look
Quoted 1 line
into the erased structures. In theroy it should not be needed because
the
Quoted 2 lines
contents of structures to be deleted should be irrelevant to kernel debugging.
So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?
I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · Reinhard Buendgen
Vivek,
I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.
So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).
Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.
When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.
Quoted 4 lines
Allowing to configure makedumpfile allows each distribution and each platform to provide appropriate filters.
I was thinking that there are very few symbols and these will not change frequently. makdumpfile is already dependent on some kenrel data strcutres and symbols for doing filtering.
But looks like you are planning to filter out lots of symbols. So in that case agreed that hardcoding is not a good idea.
Thanks
Vivek
Quoted 69 lines
Mit freundlichen Grüßen/Best Regards/Cordialement
Reinhard
Dr. Reinhard Bündgen
RAS & Crypto Architect for Linux on System z
Virtualization and Systems ManagementMail:[email protected]
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294From: Vivek Goyal <[email protected]> To: Reinhard Buendgen/Germany/IBM@IBMDECc: Dave Anderson <[email protected]>, Ananth N Mavinakayanahalli
<[email protected]>, [email protected], Mahesh J Salgaonkar <[email protected]>, "Ken'ichi Ohmichi" <[email protected]>, V Srivatsa <[email protected]>Date: 25.05.2011 21:53 Subject: Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancementto filter out kernel data from vmcore
Quoted 5 lines · Reinhard Buendgen
Hi,
to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload
within)
Quoted 3 lines
of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their
data
Quoted 4 lines
will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions
of that information sitting in some buffers to a service organization.
So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.
So atleast for above examples it does not sound as if we need symbol erase infrastructure.
Quoted 2 lines
to answer Daves concern: there is no intention that crash should ever
look
Quoted 1 line
into the erased structures. In theroy it should not be needed because
the
Quoted 2 lines
contents of structures to be deleted should be irrelevant to kernel debugging.
So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?
Quoted 9 lines
I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 12 lines · Vivek Goyal
Quoted 10 lines · Reinhard Buendgen
Vivek,
I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.
So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).
Quoted 7 lines
Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.
When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.
The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).
Quoted 98 lines
Quoted 4 lines
Allowing to configure makedumpfile allows each distribution and each platform to provide appropriate filters.
I was thinking that there are very few symbols and these will not change frequently. makdumpfile is already dependent on some kenrel data strcutres and symbols for doing filtering.
But looks like you are planning to filter out lots of symbols. So in that case agreed that hardcoding is not a good idea.
Thanks
VivekQuoted 69 lines
Mit freundlichen Grüßen/Best Regards/Cordialement
Reinhard
Dr. Reinhard Bündgen
RAS & Crypto Architect for Linux on System z
Virtualization and Systems ManagementMail:[email protected]
Phone: ++49-(0)7031-16-1130
Fax: ++49-(0)7031-16-3456IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294From: Vivek Goyal <[email protected]> To: Reinhard Buendgen/Germany/IBM@IBMDECc: Dave Anderson <[email protected]>, Ananth N Mavinakayanahalli
<[email protected]>, [email protected], Mahesh J Salgaonkar <[email protected]>, "Ken'ichi Ohmichi" <[email protected]>, V Srivatsa <[email protected]>Date: 25.05.2011 21:53 Subject: Re: [PATCH v2 0/8] makedumpfile: makedumpfile enhancementto filter out kernel data from vmcore
Quoted 5 lines · Reinhard Buendgen
Hi,
to answer Vivek questions first: Eventually we want to be able to erase all data that a customer may consider sensitive to her privacy. In addition to encryption key that may be the contents (i.e. payload
within)
Quoted 3 lines
of all kinds of I/O buffers. Consider you are running a kvm based hypervisor and want its dump to be analyized while promising your customers whose guests you run on that hypervisor that none of their
data
Quoted 4 lines
will be externalized. Or consider your system reads a spreadsheat with bank account or health information. You might not want to send fractions
of that information sitting in some buffers to a service organization.
So for direct IO, buffer is still in user space and should be filtered out when we filter out user space pages using mkdumpfile. For kvm, I am assuming that all the pages belong to qemu process and once we are filtering out user space pages, any data belonging to guest will go away.
So atleast for above examples it does not sound as if we need symbol erase infrastructure.
Quoted 2 lines
to answer Daves concern: there is no intention that crash should ever
look
Quoted 1 line
into the erased structures. In theroy it should not be needed because
the
Quoted 2 lines
contents of structures to be deleted should be irrelevant to kernel debugging.
So what are those kernel structures which we are planning to delete and are irrelevant to kernel debugging by crash?
Quoted 9 lines
I think we are missing something here. If there are only few known structures we want to get rid of, lets hardcode it in makedumpfile instead of giving user a generic infrastructure. That way we know that we are not leaking information at the same time making sure that analysis tools are working.
Thanks
Vivek_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Signature · 6 lines
Mahesh J Salgaonkar _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hi Mahesh,
(I'm back to makedumpfile devel for merging this patchset,
because Tachibana-san is busy.)
Sorry for replying an old mail.
Quoted 16 lines · On Fri, 27 May 2011 12:26:21 +0530 Mahesh J Salgaonkar…
Quoted 15 lines · Vivek Goyal
Quoted 10 lines · Reinhard Buendgen
Vivek,
I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.
So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.
Quoted 8 lines
Quoted 4 lines
When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.
The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).
IIUC, there are be 2 dumpfiles on customer site by the above approach. The one is with privacy/secret data, and another is without.
If correct, I like this approach because a customer can have two options when the crash utility cannot read a dumpfile without privacy/secret data on support site.
First option:
For digging a problem, a customer sends a dumpfile with privacy/secret data to support site.
Second option:
For protecting privacy/secret data, a customer gives up digging a problem.
Thanks
Ken'ichi Ohmichi
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hi Ken'ichi,
Quoted 5 lines · Ken'ichi Ohmichi
Hi Mahesh,
(I'm back to makedumpfile devel for merging this patchset,
because Tachibana-san is busy.)
Good to see that you are back.
Quoted 26 lines
Sorry for replying an old mail.
Quoted 21 lines · On Fri, 27 May 2011 12:26:21 +0530 Mahesh J Salgaonkar…
Quoted 19 lines · Vivek Goyal
Quoted 10 lines · Reinhard Buendgen
Vivek,
I/O is not restricted to disk I/O (it may be network I/O, data sent to crtypto cards etc.) and it is not always direct, Device drivers may have buffers to which such data is copied.
So it is more than just keys, and it may change over time.
I do not think hardwiring a filter in makedumpfile is a good idea because you would need a new makedumpfile release for every Distribution (release).Ok, so we are worried about data being in slub/slab caches or driver's kmalloced buffers etc.
When do I need access to debuginfo files. I am assuming that makedumpfile reads them in first kernel sometime and stores relevant info in initramfs. Otherwise, it is not possible to get to it in second kernel after crash.
Quoted 6 lines
Quoted 3 lines
The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).
IIUC, there are be 2 dumpfiles on customer site by the above approach. The one is with privacy/secret data, and another is without.
Correct.
Quoted 4 lines
If correct, I like this approach because a customer can have two options when the crash utility cannot read a dumpfile without privacy/secret data on support site.
The crash utility would just display a warning message early on during invocation. Most of the time crash tool will be able to read/analyze the dump unless someone scrubs out the data on which crash utility is dependant on. And as I mentioned previously, this is intended just as a security filter and not to be used as detrimental to crash's analysis of the dump.
Quoted 11 lines
First option:
For digging a problem, a customer sends a dumpfile with privacy/secret data to support site.Second option:
For protecting privacy/secret data, a customer gives up digging a problem.Thanks
Ken'ichi Ohmichi
Thanks,
-Mahesh.
Quoted 5 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Signature · 6 lines
Mahesh J Salgaonkar _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hi Mahesh,
Thank you for quick response.
Quoted 15 lines · On Fri, 15 Jul 2011 12:40:29 +0530 Mahesh J Salgaonkar…
Quoted 7 lines
Quoted 4 lines
The current approach is to re-run the makedumpfile on the crash dump (generated in the second kernel) when we are back into production kernel (1st kernel).
IIUC, there are be 2 dumpfiles on customer site by the above approach. The one is with privacy/secret data, and another is without.
Correct.
Quoted 4 lines
If correct, I like this approach because a customer can have two options when the crash utility cannot read a dumpfile without privacy/secret data on support site.
Quoted 5 lines
The crash utility would just display a warning message early on during invocation. Most of the time crash tool will be able to read/analyze the dump unless someone scrubs out the data on which crash utility is dependant on. And as I mentioned previously, this is intended just as a security filter and not to be used as detrimental to crash's analysis of the dump.
I see, this feature will not be a problem in most cases. However a customer can create unreadable dumpfile by using this feature, and current makedumpfile does not have such feature.
Of course unreadable dumpfile is due to wrong makedumpfile.conf, and I like this feature anyway because a customer has options.
Thanks
Ken'ichi Ohmichi
Quoted 6 lines
Quoted 6 lines
First option:
For digging a problem, a customer sends a dumpfile with privacy/secret data to support site.Second option:
For protecting privacy/secret data, a customer gives up digging a problem.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Mahesh J Salgaonkar
Hi All,
Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).
Quoted 8 lines
Quoted 3 lines
This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.
Mahesh,
Apart from security keys, what are the other examples of data which needs to be removed?
The idea is to allow customer to remove any such confidential data that he thinks should not be disclosed outside. The framework is generic enough to accommodate any new such fields. The is intended just as a security filter and not to be used as detrimental to crash's analysis of the dump.
Quoted 6 lines
By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.
The data to be erased is poisoned with character 'X' (58 in Hex).
The last two patches 7/8 and 8/8 introduces eraseinfo section into filtered compressed kdump and ELF kdump file. The compressed kdump file now carries additional fields namely offset_eraseinfo and size_eraseinfo in kdump sub header that can help crash to identify whether filtering is been done. Similarly, ELF kdump file now contains new ELF note of type NT_ERASEINFO, that can help crash to identify whether the filtering has been done or not.
Thanks,
-Mahesh.
Quoted 62 lines
Thanks
VivekQuoted 58 lines
This feature introduces a filter config file where, using filter commands, user can specify desired kernel data symbols and it's members that need to be filtered out while creating o/p DUMPFILE. The Syntax for filter commands are provided in the
makedumpfile.conf(8)man page.The first 4 patches prepares the base work for filtering framework. The patches 5 and 6 implements the generic filtering framework to erase desired kernel data. The last two patches implements inclusion of eraseinfo data into o/p dumpfile (kdump as well as ELF). The eraseinfo data contains the effective lines from configuration files.
I have tested these patches on
x86_64and s390x architecture against RHEL6 GA kernel. The feature supports filtering data from ELF as well as kdump-compressed formatted dump.Change in v2:
- Changes in patch 5 and inclusion of new patches 7 & 8
- Re-named filter.conf file to makedumpfile.conf and some cleanup work.
- Introduced last two patches that implements inclusion of eraseinfo data into kdump compressed as well as ELF formatted dumpfile.Please review the patchset and let me know your comments.
Thanks,
-Mahesh.
---Mahesh Salgaonkar (8):
makedumpfile: Add '--config' option to specify filter config file.
makedumpfile: Apply relocation while loading module debuginfo.
makedumpfile: Load the module symbol data from vmcore.
makedumpfile: Introduce routines to get type name from debuginfo.
makedumpfile: Read and process filter commands from config file.
makedumpfile: Read and process 'for' command from config file.
makedumpfile: Add erased information in compressed kdump file
makedumpfile: Add erase information in ELF formatted dumpfileIMPLEMENTATION | 6 Makefile | 8 diskdump_mod.h | 2 makedumpfile.8 | 20 makedumpfile.c | 2747 ++++++++++++++++++++++++++++++++++++++++++++++++--- makedumpfile.conf | 149 +++ makedumpfile.conf.8 | 419 ++++++++ makedumpfile.h | 199 ++++ 8 files changed, 3394 insertions(+), 156 deletions(-) create mode 100644 makedumpfile.conf create mode 100644 makedumpfile.conf.8Signature · 6 lines
Signature _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Signature · 6 lines
Mahesh J Salgaonkar _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 13 lines · Mahesh J Salgaonkar
Quoted 12 lines · Vivek Goyal
Quoted 10 lines · Mahesh J Salgaonkar
Hi All,
Please find the version 2 of makedumpfile enhancement patchset that introduces a data filtering feature which enables makedumpfile to filter out desired kernel symbol data and it's members from the specified VMCORE file. The data to be filtered out is poisoned with character 'X' (58 in Hex).
This feature will be very useful for the customers who wants to erase the customer sensitive data like security keys and other confidential data, in DUMPFILE before sending it to support team for analysis.
Quoted 8 lines
Quoted 4 lines
Mahesh,
Apart from security keys, what are the other examples of data which needs to be removed?
The idea is to allow customer to remove any such confidential data that he thinks should not be disclosed outside. The framework is generic enough to accommodate any new such fields.
So what is that info which customer whould like to erase (apart from security keys?) Do we have some more concrete examples?
Quoted 3 lines
The is intended just as a
security filter and not to be used as detrimental to crash's analysis of the dump.
How would we know in advance that what symbols crash is dependent on to even open crash dump and we are not erasing one of those symbols. In other mail Dave mentioned two symbols (vmlist and one more) which crash is dependent on and we seem to be clearing that.
I am worried about filtering out symbols which make crash non functional. Dave mentioned that one can remove the pointer which contains the list of modules etc. I am worried what if module caused the crash and we break the chain in such a way that reaching to module data structures is not possible anymore.
If security key is only example, should we just hardcode that in makedumpfile to wipe that info instead of creating a generic framework for this. If we give user too much flexibility they might delete way too many symbols to be safe and that might result in un-analyzable crash dump.
Quoted 10 lines
Quoted 7 lines
By erasing the data, you mean you set it to zero? Will it make sense to poison it with some specific pattern so that if crash or other tool are looking into it they know it has been posioned and it is not a case of corruption. This might help if a user erased a symbol which was required by crash for some analysis etc.
The data to be erased is poisoned with character 'X' (58 in Hex).
Quoted 7 lines
The last two patches 7/8 and 8/8 introduces eraseinfo section into filtered compressed kdump and ELF kdump file. The compressed kdump file now carries additional fields namely
offset_eraseinfoandsize_eraseinfoin kdump sub header that can help crash to identify whether filtering is been done. Similarly, ELF kdump file now contains new ELF note of typeNT_ERASEINFO, that can help crash to identify whether the filtering has been done or not.
And how this information is useful to crash? I mean why would crash care whether filtering has been done or not and if filtering has been done then do what?
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
----- Original Message -----
Quoted 11 lines
Quoted 10 lines
The data to be erased is poisoned with character 'X' (58 in Hex).
The last two patches 7/8 and 8/8 introduces eraseinfo section into filtered compressed kdump and ELF kdump file. The compressed kdump file now carries additional fields namely
offset_eraseinfoandsize_eraseinfoin kdump sub header that can help crash to identify whether filtering is been done. Similarly, ELF kdump file now contains new ELF note of typeNT_ERASEINFO, that can help crash to identify whether the filtering has been done or not.
Quoted 3 lines
And how this information is useful to crash? I mean why would crash care whether filtering has been done or not and if filtering has been done then do what?
It's not particularly useful to crash per se -- but rather it would be useful to a user trying to analyze the dumpfile. If a prominent warning message is displayed early on during invocation, then it won't be a big surprise if/when things are subsequently broken/crippled.
Dave
Quoted 2 lines
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec