kdump on Xen hypervisor and guests, more tests for utilities, like makedumpfile, mkdumprd, kexec etc

Hi,

Cai has proposed to work on the above LTP-KDUMP test cases enrichment/enhancements. Please let us know about your views on the same. We encourage people to review his proposal and the corresponding upcoming test cases. I am going to put this soon on the LTP-KDUMP plan document.

http://ltp.cvs.sourceforge.net/ltp/ltp/testcases/kdump/doc/TEST_PLAN.txt,

..................................

Here is my first draft plan of Kexec/Kdump tests enhancement sorted by priorities. I would like to add them as many as possible.

== filtered vmcore utilities ==
- in different compressed levels, verify the vmcore with the correct layout.
- verify it in flat file or ELF formats from a network host.

== analyse vmcore utilities ==
- GDB
- crash with better error detecting.
- crash to analyse Hypervisor and Dom0 Kernel.

== test scripts ==
- timestamp information for crash was triggered, vmcore was generated, and vmcore was verified.
- aim to 100% automation, and reduce manual setup.
- tidy up scripts.

== crash scenarios ==
- SDINT switch for ia64 if possible.
- Hypervisor crash for Virtualization.
- crashes on full- and para-virt guests.

== fix bugs in existing tests ==
- printk LKDTM module can hang the second Kernel.

== kdump configurations and init script ==
- capture vmcore after init runs.
- rpm pre- and post-scripts
- kdump_pre and kdump_post directives

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Quoted 10 lines

Hi,

Cai has proposed to work on the above LTP-KDUMP test cases enrichment/enhancements. Please let us know about your views on the same. We encourage people to review his proposal and the corresponding upcoming test cases. I am going to put this soon on the LTP-KDUMP plan document.

http://ltp.cvs.sourceforge.net/ltp/ltp/testcases/kdump/doc/TEST_PLAN.txt,

Hi Subrata/Cai,

That's a very good idea. We need to increase kdump test coverage and automate the whole thing.

Quoted 26 lines

..................................

Here is my first draft plan of Kexec/Kdump tests enhancement sorted by priorities. I would like to add them as many as possible.

== filtered vmcore utilities ==
- in different compressed levels, verify the vmcore with the correct layout.
- verify it in flat file or ELF formats from a network host.

== analyse vmcore utilities ==
- GDB
- crash with better error detecting.
- crash to analyse Hypervisor and Dom0 Kernel.

== test scripts ==
- timestamp information for crash was triggered, vmcore was generated, and vmcore was verified.
- aim to 100% automation, and reduce manual setup.
- tidy up scripts.

== crash scenarios ==
- SDINT switch for ia64 if possible.
- Hypervisor crash for Virtualization.
- crashes on full- and para-virt guests.

Quoted 8 lines

== fix bugs in existing tests ==
- printk LKDTM module can hang the second Kernel.

== kdump configurations and init script ==
- capture vmcore after init runs.
- rpm pre- and post-scripts
- kdump_pre and kdump_post directives

Can we boost the priority of this item. Making sure all the kdump config options are working as stated. This is the interface a kdump user first sees and if it does not work, then it leaves a very bad impression.

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

Thanks
Vivek

Yes. Initally, I put this item to a relative low priority, partly because kdump config options and init scripts are tend to be distro-specific, so it won't be easy to write portable tests for different distros. In addition, lots of config options are not easy to be tested automately, like raw disk target, vfat disk target, and network target etc, as testers have to setup those name manually. But, you are right, those are high priority tests for kexec/kdump in distro release, we tested most of those options manually for RHEL anyway and we had some automated tests of it, which I'll try to submit to LTP as many as possible. For those manual tests, I'll also try to find some ways to automate them. For example, for different dump targets, it is possible to verify the generated initrd file as expected.

Quoted 8 lines

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

That isn't something I have not thought of. For RHEL release testing, we will have a workflow to run those tests on any storage/network drivers, and it will report back testing results and driver matrix. However, this workflow is very distro-specific, and depends on the test farm it is using, so it does not make any sense to put it here.

Quoted 4 lines

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

That sounds like a good idea to me.

Quoted 4 lines

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

This is something could be done. I'll have a look at it.

Thanks,
CaiQian

Quoted 2 lines

Thanks
Vivek

Hi Cai,

Some patches in Bits and Pieces regarding this nearby ?

Regards--
Subrata

Quoted 41 lines

Yes. Initally, I put this item to a relative low priority, partly because kdump config options and init scripts are tend to be distro-specific, so it won't be easy to write portable tests for different distros. In addition, lots of config options are not easy to be tested automately, like raw disk target, vfat disk target, and network target etc, as testers have to setup those name manually. But, you are right, those are high priority tests for kexec/kdump in distro release, we tested most of those options manually for RHEL anyway and we had some automated tests of it, which I'll try to submit to LTP as many as possible. For those manual tests, I'll also try to find some ways to automate them. For example, for different dump targets, it is possible to verify the generated initrd file as expected.

Quoted 8 lines

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

That isn't something I have not thought of. For RHEL release testing, we will have a workflow to run those tests on any storage/network drivers, and it will report back testing results and driver matrix. However, this workflow is very distro-specific, and depends on the test farm it is using, so it does not make any sense to put it here.

Quoted 4 lines

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

That sounds like a good idea to me.

Quoted 4 lines

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

Quoted 7 lines

This is something could be done. I'll have a look at it.

Thanks,
CaiQian

Quoted 2 lines

Thanks
Vivek

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Cai,

Any headway in this front ?

Regards--
Subrata

Quoted 66 lines · Subrata Modak

Hi Cai,

Some patches in Bits and Pieces regarding this nearby ?

Regards--
Subrata

Quoted 41 lines

Yes. Initally, I put this item to a relative low priority, partly because kdump config options and init scripts are tend to be distro-specific, so it won't be easy to write portable tests for different distros. In addition, lots of config options are not easy to be tested automately, like raw disk target, vfat disk target, and network target etc, as testers have to setup those name manually. But, you are right, those are high priority tests for kexec/kdump in distro release, we tested most of those options manually for RHEL anyway and we had some automated tests of it, which I'll try to submit to LTP as many as possible. For those manual tests, I'll also try to find some ways to automate them. For example, for different dump targets, it is possible to verify the generated initrd file as expected.

Quoted 8 lines

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

That isn't something I have not thought of. For RHEL release testing, we will have a workflow to run those tests on any storage/network drivers, and it will report back testing results and driver matrix. However, this workflow is very distro-specific, and depends on the test farm it is using, so it does not make any sense to put it here.

Quoted 4 lines

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

That sounds like a good idea to me.

Quoted 4 lines

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

Quoted 7 lines

This is something could be done. I'll have a look at it.

Thanks,
CaiQian

Quoted 2 lines

Thanks
Vivek

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Hi Subrata,

From: Subrata Modak <[email protected]>
Subject: Re: [LTP] [RFC] [KDUMP] [PROPOSED WORK] kdump on Xen hypervisor and guests, more tests for utilities, like makedumpfile, mkdumprd, kexec etc
Date: Mon, 11 Aug 2008 19:02:47 +0530

Quoted 4 lines

Cai,

Any headway in this front ?

I have all test cases finished, but I am working on implementing a suitable harness to run those tests.

CaiQian

Quoted 71 lines

Regards--
Subrata

Quoted 66 lines · Subrata Modak

Hi Cai,

Some patches in Bits and Pieces regarding this nearby ?

Regards--
Subrata

Quoted 41 lines

Yes. Initally, I put this item to a relative low priority, partly because kdump config options and init scripts are tend to be distro-specific, so it won't be easy to write portable tests for different distros. In addition, lots of config options are not easy to be tested automately, like raw disk target, vfat disk target, and network target etc, as testers have to setup those name manually. But, you are right, those are high priority tests for kexec/kdump in distro release, we tested most of those options manually for RHEL anyway and we had some automated tests of it, which I'll try to submit to LTP as many as possible. For those manual tests, I'll also try to find some ways to automate them. For example, for different dump targets, it is possible to verify the generated initrd file as expected.

Quoted 8 lines

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

That isn't something I have not thought of. For RHEL release testing, we will have a workflow to run those tests on any storage/network drivers, and it will report back testing results and driver matrix. However, this workflow is very distro-specific, and depends on the test farm it is using, so it does not make any sense to put it here.

Quoted 4 lines

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

That sounds like a good idea to me.

Quoted 4 lines

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

Quoted 7 lines

This is something could be done. I'll have a look at it.

Thanks,
CaiQian

Quoted 2 lines

Thanks
Vivek

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Quoted 6 lines

Hi Subrata,

From: Subrata Modak <[email protected]>
Subject: Re: [LTP] [RFC] [KDUMP] [PROPOSED WORK] kdump on Xen hypervisor and guests, more tests for utilities, like makedumpfile, mkdumprd, kexec etc
Date: Mon, 11 Aug 2008 19:02:47 +0530

Quoted 7 lines

Quoted 4 lines

Cai,

Any headway in this front ?

I have all test cases finished, but I am working on implementing a suitable harness to run those tests.

Thanks Cai. Happy to know that you are done with all the test cases, and only the integration part is pending. I am little bit confused when you say that you are working on implementing a suitable harness to run those tests. Does that mean that the way the present LTP-Kdump tests are run is not suitable to accommodate the enhancements/additions that you are doing to the LTP-Kdump ?

I do not mind in having a nice harness within LTP for executing the LTP-Kdump tests, but i would hope that the new harness should also include the existing LTP-Kdump tests, so that you use the same (new harness) to tests everything in LTP-Kdump. Let me know what you think for the same.

Regards--
Subrata

Quoted 74 lines

CaiQian

Quoted 71 lines

Regards--
Subrata

Quoted 66 lines · Subrata Modak

Hi Cai,

Some patches in Bits and Pieces regarding this nearby ?

Regards--
Subrata

Quoted 41 lines

Yes. Initally, I put this item to a relative low priority, partly because kdump config options and init scripts are tend to be distro-specific, so it won't be easy to write portable tests for different distros. In addition, lots of config options are not easy to be tested automately, like raw disk target, vfat disk target, and network target etc, as testers have to setup those name manually. But, you are right, those are high priority tests for kexec/kdump in distro release, we tested most of those options manually for RHEL anyway and we had some automated tests of it, which I'll try to submit to LTP as many as possible. For those manual tests, I'll also try to find some ways to automate them. For example, for different dump targets, it is possible to verify the generated initrd file as expected.

Quoted 8 lines

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

That isn't something I have not thought of. For RHEL release testing, we will have a workflow to run those tests on any storage/network drivers, and it will report back testing results and driver matrix. However, this workflow is very distro-specific, and depends on the test farm it is using, so it does not make any sense to put it here.

Quoted 4 lines

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

That sounds like a good idea to me.

Quoted 4 lines

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

Quoted 7 lines

This is something could be done. I'll have a look at it.

Thanks,
CaiQian

Quoted 2 lines

Thanks
Vivek

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Hi Subrata,

From: Subrata Modak <[email protected]>
Subject: Re: [LTP] [RFC] [KDUMP] [PROPOSED WORK] kdump on Xen hypervisor and guests, more tests for utilities, like makedumpfile, mkdumprd, kexec etc
Date: Tue, 12 Aug 2008 11:43:02 +0530

Quoted 15 lines · Cai Qian

Quoted 6 lines

Hi Subrata,

From: Subrata Modak <[email protected]>
Subject: Re: [LTP] [RFC] [KDUMP] [PROPOSED WORK] kdump on Xen hypervisor and guests, more tests for utilities, like makedumpfile, mkdumprd, kexec etc
Date: Mon, 11 Aug 2008 19:02:47 +0530

Quoted 7 lines

Quoted 4 lines

Cai,

Any headway in this front ?

I have all test cases finished, but I am working on implementing a suitable harness to run those tests.

Quoted 7 lines

Thanks Cai. Happy to know that you are done with all the test cases, and only the integration part is pending. I am little bit confused when you say that you are working on implementing a suitable harness to run those tests. Does that mean that the way the present LTP-Kdump tests are run is not suitable to accommodate the enhancements/additions that you are doing to the LTP-Kdump ?

What I have already done is to create several basic building blocks for testing of Kdump. For example, setup-bare-metal, config-ext3, crash-sysrq-c, crash-lkdtm, analyse-crash etc. You probably could guess what they are doing from their names. Then, we could combine those building blocks to create test cases, which should give us more freedom and flexibility to test things. I am going to include some pre-defined test cases into LTP as well. I might propose new tests as LTP-kdump-ng.

Quoted 6 lines

I do not mind in having a nice harness within LTP for executing the LTP-Kdump tests, but i would hope that the new harness should also include the existing LTP-Kdump tests, so that you use the same (new harness) to tests everything in LTP-Kdump. Let me know what you think for the same.

The problem that I am trying to figure it out is that we need a suitable harness to schedule those tests and report results. For example, there is a test case is like the following,

setup-bare-metal
config-ext3:uuid
crash-sysrq-c
analyse-crash

The first and the third steps requires to reboot the machine, so we need a harness could resume execution of tests after reboot. I am thinking of using cron job as in existing LTP-kdump tests, or a tests injector running as a daemon, or something else.

New tests should incorporate all existing tests and coverage of Kdump in LTP.

CaiQian

Quoted 78 lines

Regards--
Subrata

Quoted 74 lines

CaiQian

Quoted 71 lines

Regards--
Subrata

Quoted 66 lines · Subrata Modak

Hi Cai,

Some patches in Bits and Pieces regarding this nearby ?

Regards--
Subrata

Quoted 41 lines

Yes. Initally, I put this item to a relative low priority, partly because kdump config options and init scripts are tend to be distro-specific, so it won't be easy to write portable tests for different distros. In addition, lots of config options are not easy to be tested automately, like raw disk target, vfat disk target, and network target etc, as testers have to setup those name manually. But, you are right, those are high priority tests for kexec/kdump in distro release, we tested most of those options manually for RHEL anyway and we had some automated tests of it, which I'll try to submit to LTP as many as possible. For those manual tests, I'll also try to find some ways to automate them. For example, for different dump targets, it is possible to verify the generated initrd file as expected.

Quoted 8 lines

Quoted 3 lines

== increase coverages for new kexec/kdump development efforts ==
- new reserved region syntax in Kernel.

Another important thing we need to focus on is driver testing. Drivers can fail to initialize in second kernel and kdump will fail. Can we do something so that we can do following.

That isn't something I have not thought of. For RHEL release testing, we will have a workflow to run those tests on any storage/network drivers, and it will report back testing results and driver matrix. However, this workflow is very distro-specific, and depends on the test farm it is using, so it does not make any sense to put it here.

Quoted 4 lines

- Collect the machine statistics on which kdump was tested and send the reports to a common place. Especially capture the storage/network driver data which can be probably be available through LTP site.

That sounds like a good idea to me.

Quoted 4 lines

- Also capture how much memory was reserved on what architecture and whether it worked or not. This will help us verify for sure that how much memory to reserve for second kernel on various architectures.

Quoted 7 lines

This is something could be done. I'll have a look at it.

Thanks,
CaiQian

Quoted 2 lines

Thanks
Vivek

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list