kernel: add support to collect hardware logs in crash recovery kernel

Add a new module crashdd that exports the /sys/kernel/crashdd/ directory in second kernel, containing collected hardware/firmware dumps.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with

requested size and invokes the device driver's registered callback function.

3. Device driver collects all hardware/firmware logs into the buffer and returns control back to crashdd.

4. Crashdd exposes the buffer as a binary file via /sys/kernel/crashdd/<driver>/<dump_file>.

Quoted 8 lines · Sat, Mar 24, 2018 at 11:56:33AM CET, rahul.lakkireddy-ut6Up61K2wZBDgjK7y7TUQ@pu…

Add a new module crashdd that exports the /sys/kernel/crashdd/ directory in second kernel, containing collected hardware/firmware dumps.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

Quoted 7 lines

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with

This smells. I need to identify the exact ASIC instance that produced the dump. To identify by driver name does not help me if I have multiple instances of the same driver. This looks wrong to me. This looks like a job for devlink where you have 1 devlink instance per 1 ASIC instance.

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

Quoted 9 lines

requested size and invokes the device driver's registered callback function.

3. Device driver collects all hardware/firmware logs into the buffer and returns control back to crashdd.

4. Crashdd exposes the buffer as a binary file via /sys/kernel/crashdd/<driver>/<dump_file>.

Quoted 22 lines · Jiri Pirko

Quoted 8 lines · Sat, Mar 24, 2018 at 11:56:33AM CET, [email protected] wrote:

Add a new module crashdd that exports the /sys/kernel/crashdd/ directory in second kernel, containing collected hardware/firmware dumps.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

Quoted 7 lines

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with

This smells. I need to identify the exact ASIC instance that produced the dump. To identify by driver name does not help me if I have multiple instances of the same driver. This looks wrong to me. This looks like a job for devlink where you have 1 devlink instance per 1 ASIC instance.

Quoted 7 lines

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

The sysfs approach proposed here had been dropped in favour exporting the dumps as ELF notes in /proc/vmcore.

Will be posting the new patches soon.

Quoted 31 lines · Rahul Lakkireddy

Quoted 29 lines · Jiri Pirko

Quoted 8 lines · Sat, Mar 24, 2018 at 11:56:33AM CET, [email protected] wrote:

Add a new module crashdd that exports the /sys/kernel/crashdd/ directory in second kernel, containing collected hardware/firmware dumps.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

Quoted 7 lines

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with

This smells. I need to identify the exact ASIC instance that produced the dump. To identify by driver name does not help me if I have multiple instances of the same driver. This looks wrong to me. This looks like a job for devlink where you have 1 devlink instance per 1 ASIC instance.

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

Quoted 4 lines

The sysfs approach proposed here had been dropped in favour exporting the dumps as ELF notes in /proc/vmcore.

Will be posting the new patches soon.

The concern was actually how you identify which device that came from. Where you read the identifier changes but sysfs or /proc/vmcore the change remains valid.

Quoted 8 lines

Quoted 4 lines

The sysfs approach proposed here had been dropped in favour exporting the dumps as ELF notes in /proc/vmcore.

Will be posting the new patches soon.

The concern was actually how you identify which device that came from. Where you read the identifier changes but sysfs or /proc/vmcore the change remains valid.

Yeah. I still don't see how you link the dump and the device.

Quoted 36 lines · Fri, Mar 30, 2018 at 08:42:00PM CEST, [email protected] wrote:

Quoted 34 lines · Rahul Lakkireddy

Quoted 22 lines · Jiri Pirko

Quoted 8 lines · Sat, Mar 24, 2018 at 11:56:33AM CET, [email protected] wrote:

Add a new module crashdd that exports the /sys/kernel/crashdd/ directory in second kernel, containing collected hardware/firmware dumps.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

Quoted 7 lines

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with

This smells. I need to identify the exact ASIC instance that produced the dump. To identify by driver name does not help me if I have multiple instances of the same driver. This looks wrong to me. This looks like a job for devlink where you have 1 devlink instance per 1 ASIC instance.

Quoted 7 lines

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

The sysfs approach proposed here had been dropped in favour exporting the dumps as ELF notes in /proc/vmcore.

Quoted 5 lines

Quoted 1 line

Will be posting the new patches soon.

The concern was actually how you identify which device that came from. Where you read the identifier changes but sysfs or /proc/vmcore the change remains valid.

Yeah. I still don't see how you link the dump and the device.

Rahul, did you look at the patchset I pointed out?

Quoted 37 lines · Jiri Pirko

Quoted 36 lines · Fri, Mar 30, 2018 at 08:42:00PM CEST, [email protected] wrote:

Quoted 34 lines · Rahul Lakkireddy

Quoted 22 lines · Jiri Pirko

Quoted 8 lines · Sat, Mar 24, 2018 at 11:56:33AM CET, [email protected] wrote:

Add a new module crashdd that exports the /sys/kernel/crashdd/ directory in second kernel, containing collected hardware/firmware dumps.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

Quoted 7 lines

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with

This smells. I need to identify the exact ASIC instance that produced the dump. To identify by driver name does not help me if I have multiple instances of the same driver. This looks wrong to me. This looks like a job for devlink where you have 1 devlink instance per 1 ASIC instance.

Quoted 7 lines

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

The sysfs approach proposed here had been dropped in favour exporting the dumps as ELF notes in /proc/vmcore.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

Will be posting the new patches soon.

The concern was actually how you identify which device that came from. Where you read the identifier changes but sysfs or /proc/vmcore the change remains valid.

Yeah. I still don't see how you link the dump and the device.

In our case, the dump and the device are being identified by the driver’s name followed by its corresponding pci bus id. I’ve posted an example in my v3 series:

https://www.spinics.net/lists/netdev/msg493781.html

Here’s an extract from the link above:

# readelf -n /proc/vmcore

Displaying notes found at file offset 0x00001000 with length 0x04003288:
Owner Data size Description
VMCOREDD_cxgb4_0000:02:00.4 0x02000fd8 Unknown note type:(0x00000700)
VMCOREDD_cxgb4_0000:04:00.4 0x02000fd8 Unknown note type:(0x00000700)

CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
CORE                 0x00000150     NT_PRSTATUS (prstatus structure)
VMCOREINFO           0x0000074f     Unknown note type: (0x00000000)

Here, for my two devices, the dump’s names are VMCOREDD_cxgb4_0000:02:00.4 and VMCOREDD_cxgb4_0000:04:00.4.

It’s really up to the callers to write their own unique name for the dump. The name is appended to “VMCOREDD_” string.

Quoted 1 line

Rahul, did you look at the patchset I pointed out?

For devlink, I think the dump name would be identified by bus_type/device_name; i.e. “pci/0000:02:00.4” for my example. Is my understanding correct?

Thanks,
Rahul

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

Quoted 5 lines

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

Hi Jiri

https://lkml.org/lkml/2018/3/20/436

How well does this API work for a 2Gbyte snapshot?

Andrew

Quoted 3 lines · Fri, Mar 30, 2018 at 05:11:29PM CEST, [email protected] wrote:

Quoted 3 lines

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

Quoted 7 lines

Quoted 2 lines

I bevieve that the solution in the patchset could be used for your usecase too.

Hi Jiri

https://lkml.org/lkml/2018/3/20/436

How well does this API work for a 2Gbyte snapshot?

Ccing Alex who did the tests.

Quoted 1 line

Andrew

Quoted 8 lines · Jiri Pirko

Quoted 7 lines · Fri, Mar 30, 2018 at 05:11:29PM CEST, [email protected] wrote:

Quoted 5 lines

Please see:
http://patchwork.ozlabs.org/project/netdev/list/?series=36524

I bevieve that the solution in the patchset could be used for your usecase too.

Hi Jiri

Quoted 4 lines

Quoted 3 lines

https://lkml.org/lkml/2018/3/20/436

How well does this API work for a 2Gbyte snapshot?

Ccing Alex who did the tests.

I didn't check the performance for such a large snapshot. From my measurement it takes 0.09s for 1 MB of data this means about ~3m.

Register callback to collect hardware/firmware dumps in second kernel before hardware/firmware is initialized. The dumps for each device will be available under /sys/kernel/crashdd/cxgb4/ directory in second kernel.

Signed-off-by Rahul Lakkireddy <[email protected]>

Signed-off-by Ganesh Goudar <[email protected]>

Register callback to collect hardware/firmware dumps in second kernel before hardware/firmware is initialized. The dumps for each device will be available under /sys/kernel/crashdd/cxgb4/ directory in second kernel.

Signed-off-by Rahul Lakkireddy <[email protected]>

Signed-off-by Ganesh Goudar <[email protected]>

---
v2:
- No Changes.

Changes since rfc v2:
- Update comments and commit message for sysfs change.

rfc v2:
- Updated dump registration to the new API in patch 1.

drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +3 -0

index e880be8e3c45..265cb026f868 100644
@@ -5527,6 +5527,18 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (err)
 		goto out_free_adapter;
 
+	if (is_kdump_kernel()) {
+		/* Collect hardware state and append to
+		 * /sys/kernel/crashdd/cxgb4/ directory

drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +0 -0

 	if (!is_t4(adapter->params.chip)) {
 		s_qpp = (QUEUESPERPAGEPF0_S +

Signature · 1 line

2.14.1

Quoted 16 lines · Rahul Lakkireddy

Register callback to collect hardware/firmware dumps in second kernel before hardware/firmware is initialized. The dumps for each device will be available under /sys/kernel/crashdd/cxgb4/ directory in second kernel.

Signed-off-by Rahul Lakkireddy <[email protected]>

Signed-off-by Ganesh Goudar <[email protected]>

---
v2:
- No Changes.

Changes since rfc v2:
- Update comments and commit message for sysfs change.

rfc v2:
- Updated dump registration to the new API in patch 1.

[...]

Quoted 18 lines

drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +10 -0

index e880be8e3c45..265cb026f868 100644
@@ -5527,6 +5527,18 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (err)
 		goto out_free_adapter;
 
+	if (is_kdump_kernel()) {
+		/* Collect hardware state and append to
+		 * /sys/kernel/crashdd/cxgb4/ directory
+		 */
+		err = cxgb4_cudbg_crashdd_add_dump(adapter);
+		if (err) {
+			dev_warn(adapter->pdev_dev,
+				 "Fail collecting crash device dump, err: %d. Continuing\n",
+				 err);
+			err = 0;

drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · line 5542

Quoted 3 lines

drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +2 -0

+		}
+	}
 

The problem I see with this approach is that you require that the driver is built into the kdump kernel (or present as a module in the kdump initramfs), and that you will probe the device during the collection of the dumps.

IMHO, if you are going to require the device to be probed by the same driver during kdump, you might just as well use the device object itself to present the crash data. I think that's what Stephen Hemminger meant when he said to use sysfs. No need at all for any special crashdd. Just add an attribute or attribute group to the device object.

Otherwise, as Eric Biederman pointed out, you should just add that data into the vmcore before you kexec, so you don't even need to look at a different file, and the driver does not even need to be present in the kdump kernel.

On production servers running variety of workloads over time, kernel panic can happen sporadically after days or even months. It is important to collect as much debug logs as possible to root cause and fix the problem, that may not be easy to reproduce. Snapshot of underlying hardware/firmware state (like register dump, firmware logs, adapter memory, etc.), at the time of kernel panic will be very helpful while debugging the culprit device driver.

This series of patches add new generic framework that enable device drivers to collect device specific snapshot of the hardware/firmware state of the underlying device in the crash recovery kernel. In crash recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/ directory, which is copied by user space scripts for post-analysis.

A kernel module crashdd is newly added. In crash recovery kernel, crashdd exposes /sys/kernel/crashdd/ directory containing device specific hardware/firmware logs.

The sequence of actions done by device drivers to append their device specific hardware/firmware logs to /sys/kernel/crashdd/ directory are as follows:

1. During probe (before hardware is initialized), device drivers register to the crashdd module (via crashdd_add_dump()), with callback function, along with buffer size and log name needed for firmware/hardware log collection.

2. Crashdd creates a driver's directory under /sys/kernel/crashdd/<driver>. Then, it allocates the buffer with requested size and invokes the device driver's registered callback function.

3. Device driver collects all hardware/firmware logs into the buffer and returns control back to crashdd.

4. Crashdd exposes the buffer as a file via
/sys/kernel/crashdd/<driver>/<dump_file>.

5. User space script (/usr/lib/kdump/kdump-lib-initramfs.sh) copies the entire /sys/kernel/crashdd/ directory to /var/crash/ directory.

Patch 1 adds crashdd module to allow drivers to register callback to collect the device specific hardware/firmware logs. The module also exports /sys/kernel/crashdd/ directory containing the hardware/firmware logs.

Patch 2 shows a cxgb4 driver example using the API to collect hardware/firmware logs in crash recovery kernel, before hardware is initialized. The logs for the devices are made available under /sys/kernel/crashdd/cxgb4/ directory.

Thanks,
Rahul

RFC v1: https://lkml.org/lkml/2018/3/2/542
RFC v2: https://lkml.org/lkml/2018/3/16/326

---
v2:
- Added ABI Documentation for crashdd.
- Directly use octal permission instead of macro.

Changes since rfc v2:
- Moved exporting crashdd from procfs to sysfs. Suggested by Stephen Hemminger <[email protected]>
- Moved code from fs/proc/crashdd.c to fs/crashdd/ directory.
- Replaced all proc API with sysfs API and updated comments.
- Calling driver callback before creating the binary file under crashdd sysfs.
- Changed binary dump file permission from S_IRUSR to S_IRUGO.
- Changed module name from CRASH_DRIVER_DUMP to CRASH_DEVICE_DUMP.

rfc v2:
- Collecting logs in 2nd kernel instead of during kernel panic. Suggested by Eric Biederman <[email protected]>.
- Added new crashdd module that exports /proc/crashdd/ containing driver's registered hardware/firmware logs in patch 1.
- Replaced the API to allow drivers to register their hardware/firmware log collect routine in crash recovery kernel in patch 1.
- Updated patch 2 to use the new API in patch 1.

Rahul Lakkireddy (2):
fs/crashdd: add API to collect hardware dump in second kernel
cxgb4: collect hardware dump in second kernel

 Documentation/ABI/testing/sysfs-kernel-crashdd   |  34 ++++
 drivers/net/ethernet/chelsio/cxgb4/cxgb4.h       |   4 +
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_cudbg.c |  25 +++
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_cudbg.h |   3 +
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c  |  12 ++
 fs/Kconfig                                       |   1 +
 fs/Makefile                                      |   1 +
 fs/crashdd/Kconfig                               |  10 +
 fs/crashdd/Makefile                              |   3 +
 fs/crashdd/crashdd.c                             | 233 +++++++++++++++++++++++
 fs/crashdd/crashdd_internal.h                    |  24 +++
 include/linux/crashdd.h                          |  24 +++
 12 files changed, 374 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-crashdd
 create mode 100644 fs/crashdd/Kconfig
 create mode 100644 fs/crashdd/Makefile
 create mode 100644 fs/crashdd/crashdd.c
 create mode 100644 fs/crashdd/crashdd_internal.h
 create mode 100644 include/linux/crashdd.h

Quoted 14 lines · Rahul Lakkireddy

On production servers running variety of workloads over time, kernel panic can happen sporadically after days or even months. It is important to collect as much debug logs as possible to root cause and fix the problem, that may not be easy to reproduce. Snapshot of underlying hardware/firmware state (like register dump, firmware logs, adapter memory, etc.), at the time of kernel panic will be very helpful while debugging the culprit device driver.

This series of patches add new generic framework that enable device drivers to collect device specific snapshot of the hardware/firmware state of the underlying device in the crash recovery kernel. In crash recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/ directory, which is copied by user space scripts for post-analysis.

Quoted 3 lines

A kernel module crashdd is newly added. In crash recovery kernel, crashdd exposes /sys/kernel/crashdd/ directory containing device specific hardware/firmware logs.

Have you looked at instead of adding a sysfs file adding the dumps as additional elf notes in /proc/vmcore?

That should allow existing tools to capture your extended dump information with no code changes, and it will allow having a single file core dump for storing the information.

Both of which should mean something that will integrate better into existing flows.

The interface logic of the driver should be essentially the same.

Also have you tested this and seen how well your current logic captures the device information?

Quoted 17 lines · Eric W. Biederman

Quoted 14 lines · Rahul Lakkireddy

On production servers running variety of workloads over time, kernel panic can happen sporadically after days or even months. It is important to collect as much debug logs as possible to root cause and fix the problem, that may not be easy to reproduce. Snapshot of underlying hardware/firmware state (like register dump, firmware logs, adapter memory, etc.), at the time of kernel panic will be very helpful while debugging the culprit device driver.

This series of patches add new generic framework that enable device drivers to collect device specific snapshot of the hardware/firmware state of the underlying device in the crash recovery kernel. In crash recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/ directory, which is copied by user space scripts for post-analysis.

Quoted 7 lines

Quoted 3 lines

A kernel module crashdd is newly added. In crash recovery kernel, crashdd exposes /sys/kernel/crashdd/ directory containing device specific hardware/firmware logs.

Have you looked at instead of adding a sysfs file adding the dumps as additional elf notes in /proc/vmcore?

I see the crash recovery kernel's memory is not present in any of the the PT_LOAD headers. So, makedumpfile is not collecting the dumps that are in crash recovery kernel's memory.

Also, are you suggesting exporting the dumps themselves as PT_NOTE instead? I'll look into doing it this way.

Quoted 30 lines · Rahul Lakkireddy

Quoted 17 lines · Eric W. Biederman

Quoted 14 lines · Rahul Lakkireddy

On production servers running variety of workloads over time, kernel panic can happen sporadically after days or even months. It is important to collect as much debug logs as possible to root cause and fix the problem, that may not be easy to reproduce. Snapshot of underlying hardware/firmware state (like register dump, firmware logs, adapter memory, etc.), at the time of kernel panic will be very helpful while debugging the culprit device driver.

This series of patches add new generic framework that enable device drivers to collect device specific snapshot of the hardware/firmware state of the underlying device in the crash recovery kernel. In crash recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/ directory, which is copied by user space scripts for post-analysis.

Quoted 7 lines

Quoted 3 lines

A kernel module crashdd is newly added. In crash recovery kernel, crashdd exposes /sys/kernel/crashdd/ directory containing device specific hardware/firmware logs.

Have you looked at instead of adding a sysfs file adding the dumps as additional elf notes in /proc/vmcore?

I see the crash recovery kernel's memory is not present in any of the the PT_LOAD headers. So, makedumpfile is not collecting the dumps that are in crash recovery kernel's memory.

Quoted 2 lines

Also, are you suggesting exporting the dumps themselves as PT_NOTE instead? I'll look into doing it this way.

Yes. I was suggesting exporting the dumps themselves as PT_NOTE in /proc/vmcore. I think that will allow makedumpfile to collect your new information without modification.

Quoted 35 lines · Eric W. Biederman

Quoted 32 lines · Rahul Lakkireddy

Quoted 17 lines · Eric W. Biederman

Quoted 14 lines · Rahul Lakkireddy

On production servers running variety of workloads over time, kernel panic can happen sporadically after days or even months. It is important to collect as much debug logs as possible to root cause and fix the problem, that may not be easy to reproduce. Snapshot of underlying hardware/firmware state (like register dump, firmware logs, adapter memory, etc.), at the time of kernel panic will be very helpful while debugging the culprit device driver.

This series of patches add new generic framework that enable device drivers to collect device specific snapshot of the hardware/firmware state of the underlying device in the crash recovery kernel. In crash recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/ directory, which is copied by user space scripts for post-analysis.

Quoted 7 lines

Quoted 3 lines

A kernel module crashdd is newly added. In crash recovery kernel, crashdd exposes /sys/kernel/crashdd/ directory containing device specific hardware/firmware logs.

Have you looked at instead of adding a sysfs file adding the dumps as additional elf notes in /proc/vmcore?

I see the crash recovery kernel's memory is not present in any of the the PT_LOAD headers. So, makedumpfile is not collecting the dumps that are in crash recovery kernel's memory.

Also, are you suggesting exporting the dumps themselves as PT_NOTE instead? I'll look into doing it this way.

Quoted 4 lines

Yes. I was suggesting exporting the dumps themselves as PT_NOTE in /proc/vmcore. I think that will allow makedumpfile to collect your new information without modification.

If I export the dumps themselves as PT_NOTE in /proc/vmcore, can the crash tool work without modification; i.e can crash tool extract these notes?