drm/radeon kexec fixes
Subsystems (5)
cover drm/radeon kexec fixes
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.
Markus Trippelsdorf (3):
kexec: get rid of late printk
drm/radeon: Implement radeon_pci_shutdown
drm/radeon: get rid of r100_restore_sanity hack
drivers/gpu/drm/radeon/r100.c | 27 --------------------------- drivers/gpu/drm/radeon/r300.c | 2 -- drivers/gpu/drm/radeon/r420.c | 2 -- drivers/gpu/drm/radeon/r520.c | 2 -- drivers/gpu/drm/radeon/radeon_asic.h | 1 - drivers/gpu/drm/radeon/radeon_drv.c | 10 ++++++++++ drivers/gpu/drm/radeon/rs400.c | 2 -- drivers/gpu/drm/radeon/rs600.c | 2 -- drivers/gpu/drm/radeon/rs690.c | 2 -- drivers/gpu/drm/radeon/rv515.c | 2 -- kernel/kexec.c | 1 - 11 files changed, 10 insertions(+), 43 deletions(-)
Signature · 1 line
Markus
PATCH 1/3 kexec: get rid of late printk
kexec calls:
printk(KERN_EMERG "Starting new kernel\n");
late before calling machine_shutdown().
However at this point the underlying fb device may have already been shutdown. This causes the kernel to hang.
Fix by simply getting rid of the printk call.
--- kernel/kexec.c | 1 - 1 file changed, 1 deletion(-)
kernel/kexec.c +0 -1
@@ -1679,7 +1679,6 @@ int kernel_kexec(void)#endif {kernel_restart_prepare (NULL ); -printk (KERN_EMERG "Starting new kernel\n" );machine_shutdown (); }
Signature · 1 line
Markus
PATCH 2/3 drm/radeon: Implement radeon_pci_shutdown
In reply to Markus TrippelsdorfCurrently radeon devices are not properbly shutdown during kexec. This cases a varity of issues, e.g. dpm initialization failures. Fix this by implementing a radeon_pci_shutdown function, that unloads the driver cleanly.
--- drivers/gpu/drm/radeon/radeon_drv.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/gpu/drm/radeon/radeon_drv.c +10 -0
@@ -380,6 +380,15 @@ static const struct file_operations radeon_driver_kms_fops = {#endif }; + +static void +radeon_pci_shutdown (struct pci_dev *pdev) +{ +struct drm_device *dev =pci_get_drvdata (pdev); + +radeon_driver_unload_kms (dev); +} +static struct drm_driver kms_driver = { .driver_features = DRIVER_USE_AGP |
@@ -453,6 +462,7 @@ static struct pci_driver radeon_kms_pci_driver = { .remove = radeon_pci_remove, .suspend = radeon_pci_suspend, .resume = radeon_pci_resume, + .shutdown = radeon_pci_shutdown, };static int __init radeon_init (void )
Signature · 1 line
Markus
PATCH 3/3 drm/radeon: get rid of r100_restore_sanity hack
In reply to Markus TrippelsdorfNow that radeon devices are properly shutdown during kexec, we can get rid of r100_restore_sanity.
--- drivers/gpu/drm/radeon/r100.c | 27 --------------------------- drivers/gpu/drm/radeon/r300.c | 2 -- drivers/gpu/drm/radeon/r420.c | 2 -- drivers/gpu/drm/radeon/r520.c | 2 -- drivers/gpu/drm/radeon/radeon_asic.h | 1 - drivers/gpu/drm/radeon/rs400.c | 2 -- drivers/gpu/drm/radeon/rs600.c | 2 -- drivers/gpu/drm/radeon/rs690.c | 2 -- drivers/gpu/drm/radeon/rv515.c | 2 -- 9 files changed, 42 deletions(-)
drivers/gpu/drm/radeon/r100.c +0 -27
@@ -3938,31 +3938,6 @@ void r100_fini(struct radeon_device *rdev) rdev->bios =NULL ; } -/* -* Due to how kexec works, it can leave the hw fully initialised when it -* boots the new kernel. However doing our init sequence with the CP and -* WB stuff setup causes GPU hangs on the RN50 at least. So at startup -* do some quick sanity checks and restore sane values to avoid this -* problem. -*/ -void r100_restore_sanity (struct radeon_device *rdev) -{ -u32 tmp; - - tmp =RREG32 (RADEON_CP_CSQ_CNTL ); -if (tmp) { -WREG32 (RADEON_CP_CSQ_CNTL ,0 ); - } - tmp =RREG32 (RADEON_CP_RB_CNTL ); -if (tmp) { -WREG32 (RADEON_CP_RB_CNTL ,0 ); - } - tmp =RREG32 (RADEON_SCRATCH_UMSK ); -if (tmp) { -WREG32 (RADEON_SCRATCH_UMSK ,0 ); - } -} -int r100_init (struct radeon_device *rdev) {int r;
@@ -3975,8 +3950,6 @@ int r100_init(struct radeon_device *rdev)radeon_scratch_init (rdev);/* Initialize surface registers */ radeon_surface_init (rdev); -/* sanity check some register to avoid hangs like after kexec */ -r100_restore_sanity (rdev);/* TODO: disable VGA need to use VGA request */ /* BIOS*/ if (!radeon_get_bios (rdev)) {
drivers/gpu/drm/radeon/r300.c +0 -2
@@ -1480,8 +1480,6 @@ int r300_init(struct radeon_device *rdev)/* Initialize surface registers */ radeon_surface_init (rdev);/* TODO: disable VGA need to use VGA request */ -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* BIOS*/ if (!radeon_get_bios (rdev)) {if (ASIC_IS_AVIVO (rdev))
drivers/gpu/drm/radeon/r420.c +0 -2
@@ -371,8 +371,6 @@ int r420_init(struct radeon_device *rdev)/* Initialize surface registers */ radeon_surface_init (rdev);/* TODO: disable VGA need to use VGA request */ -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* BIOS*/ if (!radeon_get_bios (rdev)) {if (ASIC_IS_AVIVO (rdev))
drivers/gpu/drm/radeon/r520.c +0 -2
@@ -256,8 +256,6 @@ int r520_init(struct radeon_device *rdev)radeon_scratch_init (rdev);/* Initialize surface registers */ radeon_surface_init (rdev); -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* TODO: disable VGA need to use VGA request */ /* BIOS*/ if (!radeon_get_bios (rdev)) {
drivers/gpu/drm/radeon/radeon_asic.h +0 -1
@@ -122,7 +122,6 @@ void r100_mc_resume(struct radeon_device *rdev, struct r100_mc_save *save);void r100_vram_init_sizes (struct radeon_device *rdev);int r100_cp_reset (struct radeon_device *rdev);void r100_vga_render_disable (struct radeon_device *rdev); -void r100_restore_sanity (struct radeon_device *rdev);int r100_cs_track_check_pkt3_indx_buffer (struct radeon_cs_parser *p,struct radeon_cs_packet *pkt,struct radeon_bo *robj);
drivers/gpu/drm/radeon/rs400.c +0 -2
@@ -510,8 +510,6 @@ int rs400_init(struct radeon_device *rdev)/* Initialize surface registers */ radeon_surface_init (rdev);/* TODO: disable VGA need to use VGA request */ -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* BIOS*/ if (!radeon_get_bios (rdev)) {if (ASIC_IS_AVIVO (rdev))
drivers/gpu/drm/radeon/rs600.c +0 -2
@@ -1018,8 +1018,6 @@ int rs600_init(struct radeon_device *rdev)radeon_scratch_init (rdev);/* Initialize surface registers */ radeon_surface_init (rdev); -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* BIOS */ if (!radeon_get_bios (rdev)) {if (ASIC_IS_AVIVO (rdev))
drivers/gpu/drm/radeon/rs690.c +0 -2
@@ -789,8 +789,6 @@ int rs690_init(struct radeon_device *rdev)radeon_scratch_init (rdev);/* Initialize surface registers */ radeon_surface_init (rdev); -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* TODO: disable VGA need to use VGA request */ /* BIOS*/ if (!radeon_get_bios (rdev)) {
drivers/gpu/drm/radeon/rv515.c +0 -2
@@ -627,8 +627,6 @@ int rv515_init(struct radeon_device *rdev)/* Initialize surface registers */ radeon_surface_init (rdev);/* TODO: disable VGA need to use VGA request */ -/* restore some register to sane defaults */ -r100_restore_sanity (rdev);/* BIOS*/ if (!radeon_get_bios (rdev)) {if (ASIC_IS_AVIVO (rdev))
Signature · 1 line
Markus
Re: [PATCH 1/3] kexec: get rid of late printk
In reply to Markus TrippelsdorfQuoted 8 lines · Markus Trippelsdorf
kexec calls:
printk(KERN_EMERG "Starting new kernel\n");
late before callingmachine_shutdown().
However at this point the underlying fb device may have already been shutdown. This causes the kernel to hang.
Fix by simply getting rid of the printk call.Signed-off-by Markus Trippelsdorf <[email protected]>
Shouldn't this be taken care of with the suspend/resume_console calls? At least that's my understanding how it works in the suspend/hibernate code, maybe kexec needs similar treatment ...
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 1/3] kexec: get rid of late printk
In reply to a message not in the archiveQuoted 9 lines · On Sun, 08 September 2013 Daniel Vetter <daniel-/[email protected]>…
Quoted 7 lines · Markus Trippelsdorf
kexec calls:
printk(KERN_EMERG "Starting new kernel\n");
late before callingmachine_shutdown().
However at this point the underlying fb device may have already been shutdown. This causes the kernel to hang.
Fix by simply getting rid of the printk call.
Quoted 5 lines
Quoted 1 line
Signed-off-by Markus Trippelsdorf <[email protected]>Shouldn't this be taken care of with the suspend/resume_console calls? At least that's my understanding how it works in the suspend/hibernate code, maybe kexec needs similar treatment ...
Is it suspend/resume_console? Shouldn't the fbcon be short-circuited or disabled once there is no more underlying fb?
Serial console, if present, as well as netconsole if network device is still alive should continue working I would say.
Bruno
Quoted 1 line
-Daniel
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
I don't know if you want to remove the sanity checks. They seem cheap and safe regardless. Are they expensive or ineffective? Moreover if they work a reasonable amount of the time that means that the kexec on panic case (where we don't shut anything down) can actually use the video, and that in general the driver will be more robust. I don't expect anyone much cares as kexec on panic is mostly used to just write a core file to the network, or the local disk. But if it is easy to keep that case working most of the time, why not.
Eric
Quoted 17 lines
Markus Trippelsdorf (3):
kexec: get rid of late printk
drm/radeon: Implementradeon_pci_shutdown
drm/radeon: get rid ofr100_restore_sanityhackdrivers/gpu/drm/radeon/r100.c | 27 --------------------------- drivers/gpu/drm/radeon/r300.c | 2 -- drivers/gpu/drm/radeon/r420.c | 2 -- drivers/gpu/drm/radeon/r520.c | 2 -- drivers/gpu/drm/radeon/radeon_asic.h | 1 - drivers/gpu/drm/radeon/radeon_drv.c | 10 ++++++++++ drivers/gpu/drm/radeon/rs400.c | 2 -- drivers/gpu/drm/radeon/rs600.c | 2 -- drivers/gpu/drm/radeon/rs690.c | 2 -- drivers/gpu/drm/radeon/rv515.c | 2 -- kernel/kexec.c | 1 - 11 files changed, 10 insertions(+), 43 deletions(-)
Quoted 7 lines · Eric W. Biederman
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.
Quoted 8 lines
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
OK. I will drop this patch. It doesn't seem to be necessary, because I cannot reproduce the printk related hang anymore.
Quoted 3 lines
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
Yes. But please note that even with radeon_pci_shutdown implemented, I still get ring test failures on roughly every eighth kexec boot:
[drm:r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (0xCAFEDEAD) radeon 0000:01:05.0: disabling GPU acceleration
That's definitely better than the current state of affairs, with ring test failures on every second boot. But I haven't figured out the reason for these failures yet. It's curious that once a ring test failure occurs, it will reliably fail after each kexec invocation, no matter how often repeated. Only a reboot brings the machine back to normal.
Quoted 8 lines
I don't know if you want to remove the sanity checks. They seem cheap and safe regardless. Are they expensive or ineffective? Moreover if they work a reasonable amount of the time that means that the kexec on panic case (where we don't shut anything down) can actually use the video, and that in general the driver will be more robust. I don't expect anyone much cares as kexec on panic is mostly used to just write a core file to the network, or the local disk. But if it is easy to keep that case working most of the time, why not.
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
Signature · 1 line
Markus
Am 09.09.2013 11:21, schrieb Markus Trippelsdorf:
Quoted 24 lines · Eric W. Biederman
Quoted 6 lines
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.Quoted 8 lines
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
OK. I will drop this patch. It doesn't seem to be necessary, because I cannot reproduce the printk related hang anymore.
Quoted 3 lines
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
Yes. But please note that even with
radeon_pci_shutdownimplemented, I still get ring test failures on roughly every eighth kexec boot:
Quoted 8 lines
[drm:r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (
0xCAFEDEAD) radeon 0000:01:05.0: disabling GPU accelerationThat's definitely better than the current state of affairs, with ring test failures on every second boot. But I haven't figured out the reason for these failures yet. It's curious that once a ring test failure occurs, it will reliably fail after each kexec invocation, no matter how often repeated. Only a reboot brings the machine back to normal.
The main problem here is that the AMD gfx hardware doesn't really support being reinitialized once booted (for various reasons). It's a (intended) limitation of the hardware design that you can only initialize certain blocks once every power cycle, so the whole approach actually will never work 100% reliable.
All you can hope for is that stopping the hardware while shutting down the old kernel and starting it again results in exactly the same hardware parameters (offsets, clock etc...) otherwise starting the blocks will just fail and you end up with disabled acceleration like above.
Sorry, but there isn't much we can do about this,
Christian.
Quoted 11 lines
Quoted 8 lines
I don't know if you want to remove the sanity checks. They seem cheap and safe regardless. Are they expensive or ineffective? Moreover if they work a reasonable amount of the time that means that the kexec on panic case (where we don't shut anything down) can actually use the video, and that in general the driver will be more robust. I don't expect anyone much cares as kexec on panic is mostly used to just write a core file to the network, or the local disk. But if it is easy to keep that case working most of the time, why not.
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
Quoted 45 lines · Markus Trippelsdorf
Quoted 7 lines · Eric W. Biederman
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.Quoted 8 lines
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
OK. I will drop this patch. It doesn't seem to be necessary, because I cannot reproduce the printk related hang anymore.
Quoted 3 lines
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
Yes. But please note that even with
radeon_pci_shutdownimplemented, I still get ring test failures on roughly every eighth kexec boot:[drm:r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (
0xCAFEDEAD) radeon 0000:01:05.0: disabling GPU accelerationThat's definitely better than the current state of affairs, with ring test failures on every second boot. But I haven't figured out the reason for these failures yet. It's curious that once a ring test failure occurs, it will reliably fail after each kexec invocation, no matter how often repeated. Only a reboot brings the machine back to normal.
Quoted 8 lines
I don't know if you want to remove the sanity checks. They seem cheap and safe regardless. Are they expensive or ineffective? Moreover if they work a reasonable amount of the time that means that the kexec on panic case (where we don't shut anything down) can actually use the video, and that in general the driver will be more robust. I don't expect anyone much cares as kexec on panic is mostly used to just write a core file to the network, or the local disk. But if it is easy to keep that case working most of the time, why not.
Quoted 2 lines
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
They shouldn't be necessary with a proper shutdown, but in this particular case, they are not very expensive. What is expensive is having a separate sanity check functions for all the various hw blocks to teardown everything on startup prior to starting it up in case kexec, etc. left the system in a bad state. It ends up amounting to a full tear down sequence followed by a full start up sequence every time you load the driver.
I can't really comment on the first patch, but the rest seem fine.
Alex
Re: [PATCH 2/3] drm/radeon: Implement radeon_pci_shutdown
In reply to Markus TrippelsdorfQuoted 1 line · Markus Trippelsdorf
Currently radeon devices are not properbly shutdown during kexec. This
properly
Quoted 43 lines
cases a varity of issues, e.g. dpm initialization failures. Fix this by implementing a
radeon_pci_shutdownfunction, that unloads the driver cleanly.Signed-off-by Markus Trippelsdorf <[email protected]>--- drivers/gpu/drm/radeon/radeon_drv.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/gpu/drm/radeon/radeon_drv.c +10
-0@@ -380,6 +380,15 @@ static const struct file_operations radeon_driver_kms_fops = {#endif }; + +static void +radeon_pci_shutdown (struct pci_dev *pdev) +{ +struct drm_device *dev =pci_get_drvdata (pdev); + +radeon_driver_unload_kms (dev); +} +static struct drm_driver kms_driver = { .driver_features = DRIVER_USE_AGP |@@ -453,6 +462,7 @@ static struct pci_driver radeon_kms_pci_driver = { .remove = radeon_pci_remove, .suspend = radeon_pci_suspend, .resume = radeon_pci_resume, + .shutdown = radeon_pci_shutdown, };static int __init radeon_init (void )Signature · 5 lines
Markus _______________________________________________ dri-devel mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 14 lines · Alex Deucher
Quoted 2 lines · Markus Trippelsdorf
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
They shouldn't be necessary with a proper shutdown, but in this particular case, they are not very expensive. What is expensive is having a separate sanity check functions for all the various hw blocks to teardown everything on startup prior to starting it up in case kexec, etc. left the system in a bad state. It ends up amounting to a full tear down sequence followed by a full start up sequence every time you load the driver.
Quoted 1 line
I can't really comment on the first patch, but the rest seem fine.
Let me reask the question just a little bit.
Is it the sanity checks that are expensive? Or is it the reinitialization that is triggered by the sanity checks that is expensive?
Quoted 1 line
From what Christian said in the other reply it sounds like this is a
game we will never completely win, but it would be nice to have half a chance in the kexec on panic case to have video. So I am curious to know if the checks are expensive when we are coming at hardware in a clean state.
Eric
Quoted 24 lines · Eric W. Biederman
Quoted 14 lines · Alex Deucher
Quoted 2 lines · Markus Trippelsdorf
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
They shouldn't be necessary with a proper shutdown, but in this particular case, they are not very expensive. What is expensive is having a separate sanity check functions for all the various hw blocks to teardown everything on startup prior to starting it up in case kexec, etc. left the system in a bad state. It ends up amounting to a full tear down sequence followed by a full start up sequence every time you load the driver.
Quoted 1 line
I can't really comment on the first patch, but the rest seem fine.
Let me reask the question just a little bit.
Is it the sanity checks that are expensive? Or is it the reinitialization that is triggered by the sanity checks that is expensive?
Quoted 5 lines
From what Christian said in the other reply it sounds like this is a game we will never completely win, but it would be nice to have half a chance in the kexec on panic case to have video. So I am curious to know if the checks are expensive when we are coming at hardware in a clean state.
The particular sanity checks from this patch set are not expensive, but we had previously discussed more extensive sanity checks for other aspects of the chips in prior conversations. Prior to this patch set, the hw is not torn down properly (might have been in the middle of DMA for example) when kexec happens. That's why the sanity checks were added in the first place. With this patch set, the sanity checks shouldn't be necessary.
Alex
Quoted 32 lines · Alex Deucher
Quoted 24 lines · Eric W. Biederman
Quoted 14 lines · Alex Deucher
Quoted 2 lines · Markus Trippelsdorf
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
They shouldn't be necessary with a proper shutdown, but in this particular case, they are not very expensive. What is expensive is having a separate sanity check functions for all the various hw blocks to teardown everything on startup prior to starting it up in case kexec, etc. left the system in a bad state. It ends up amounting to a full tear down sequence followed by a full start up sequence every time you load the driver.
Quoted 1 line
I can't really comment on the first patch, but the rest seem fine.
Let me reask the question just a little bit.
Is it the sanity checks that are expensive? Or is it the reinitialization that is triggered by the sanity checks that is expensive?
Quoted 5 lines
From what Christian said in the other reply it sounds like this is a game we will never completely win, but it would be nice to have half a chance in the kexec on panic case to have video. So I am curious to know if the checks are expensive when we are coming at hardware in a clean state.
Quoted 7 lines
The particular sanity checks from this patch set are not expensive, but we had previously discussed more extensive sanity checks for other aspects of the chips in prior conversations. Prior to this patch set, the hw is not torn down properly (might have been in the middle of DMA for example) when kexec happens. That's why the sanity checks were added in the first place. With this patch set, the sanity checks shouldn't be necessary.
I think you're talking past each other.
What Eric worries about is the »kexec on panic« case, where the shutdown method *isn't* called. In this case the sanity checks, that are only superfluous when the hardware was shutdown normally during kexec (the default case), may actually help. And because the checks aren't expensive, it wouldn't hurt to just leave them in place.
Signature · 1 line
Markus
Quoted 40 lines · Christian König
Am 09.09.2013 11:21, schrieb Markus Trippelsdorf:
Quoted 24 lines · Eric W. Biederman
Quoted 6 lines
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.Quoted 8 lines
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
OK. I will drop this patch. It doesn't seem to be necessary, because I cannot reproduce the printk related hang anymore.
Quoted 3 lines
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
Yes. But please note that even with
radeon_pci_shutdownimplemented, I still get ring test failures on roughly every eighth kexec boot:Quoted 8 lines
[drm:r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (
0xCAFEDEAD) radeon 0000:01:05.0: disabling GPU accelerationThat's definitely better than the current state of affairs, with ring test failures on every second boot. But I haven't figured out the reason for these failures yet. It's curious that once a ring test failure occurs, it will reliably fail after each kexec invocation, no matter how often repeated. Only a reboot brings the machine back to normal.
The main problem here is that the AMD gfx hardware doesn't really support being reinitialized once booted (for various reasons). It's a (intended) limitation of the hardware design that you can only initialize certain blocks once every power cycle, so the whole approach actually will never work 100% reliable.
Quoted 6 lines
All you can hope for is that stopping the hardware while shutting down the old kernel and starting it again results in exactly the same hardware parameters (offsets, clock etc...) otherwise starting the blocks will just fail and you end up with disabled acceleration like above.
Sorry, but there isn't much we can do about this,
I've tested this further and it turned out that if I revert commit f5d9b7f0f9 on top of my "drm/radeon: Implement radeon_pci_shutdown" patch, the initialization failures seem to go away completely.
Any idea what's going on?
Here's the patch:
drivers/gpu/drm/radeon/r600_dpm.c +2 -2
@@ -296,9 +296,9 @@ bool r600_dynamicpm_enabled(struct radeon_device *rdev)void r600_enable_sclk_control (struct radeon_device *rdev,bool enable) {if (enable) -WREG32_P (SCLK_PWRMGT_CNTL ,0 , ~SCLK_PWRMGT_OFF ); +WREG32_P (GENERAL_PWRMGT ,0 , ~SCLK_PWRMGT_OFF );else -WREG32_P (SCLK_PWRMGT_CNTL ,SCLK_PWRMGT_OFF , ~SCLK_PWRMGT_OFF ); +WREG32_P (GENERAL_PWRMGT ,SCLK_PWRMGT_OFF , ~SCLK_PWRMGT_OFF ); }void r600_enable_mclk_control (struct radeon_device *rdev,bool enable)
Signature · 1 line
Markus
Am 11.09.2013 11:01, schrieb Markus Trippelsdorf:
Quoted 45 lines · Christian König
Quoted 44 lines
Am 09.09.2013 11:21, schrieb Markus Trippelsdorf:
Quoted 24 lines · Eric W. Biederman
Quoted 6 lines
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.Quoted 8 lines
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
OK. I will drop this patch. It doesn't seem to be necessary, because I cannot reproduce the printk related hang anymore.
Quoted 3 lines
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
Yes. But please note that even with
radeon_pci_shutdownimplemented, I still get ring test failures on roughly every eighth kexec boot:Quoted 8 lines
[drm:r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (0xCAFEDEAD) radeon 0000:01:05.0: disabling GPU accelerationThat's definitely better than the current state of affairs, with ring test failures on every second boot. But I haven't figured out the reason for these failures yet. It's curious that once a ring test failure occurs, it will reliably fail after each kexec invocation, no matter how often repeated. Only a reboot brings the machine back to normal.
The main problem here is that the AMD gfx hardware doesn't really support being reinitialized once booted (for various reasons). It's a (intended) limitation of the hardware design that you can only initialize certain blocks once every power cycle, so the whole approach actually will never work 100% reliable.
All you can hope for is that stopping the hardware while shutting down the old kernel and starting it again results in exactly the same hardware parameters (offsets, clock etc...) otherwise starting the blocks will just fail and you end up with disabled acceleration like above.
Quoted 6 lines
Quoted 1 line
Sorry, but there isn't much we can do about this,
I've tested this further and it turned out that if I revert commit
f5d9b7f0f9on top of my "drm/radeon: Implementradeon_pci_shutdown" patch, the initialization failures seem to go away completely.Any idea what's going on?
Well DPM is mostly Alex domain, but if I have to guess I would say that the SCLK is gated by the hardware when the driver is unloaded and since DPM initialized only later not ungated when the driver loads again.
Quoted 8 lines
Here's the patch:
drivers/gpu/drm/radeon/r600_dpm.c +0
-0@@ -296,9 +296,9 @@ bool r600_dynamicpm_enabled(struct radeon_device *rdev)void r600_enable_sclk_control (struct radeon_device *rdev,bool enable)
drivers/gpu/drm/radeon/r600_dpm.c bool r600_dynamicpm_enabled(struct radeon_device *rdev) · line 304
Quoted 10 lines
drivers/gpu/drm/radeon/r600_dpm.c +2
-2{if (enable) -WREG32_P (SCLK_PWRMGT_CNTL ,0 , ~SCLK_PWRMGT_OFF ); +WREG32_P (GENERAL_PWRMGT ,0 , ~SCLK_PWRMGT_OFF );else -WREG32_P (SCLK_PWRMGT_CNTL ,SCLK_PWRMGT_OFF , ~SCLK_PWRMGT_OFF ); +WREG32_P (GENERAL_PWRMGT ,SCLK_PWRMGT_OFF , ~SCLK_PWRMGT_OFF ); }void r600_enable_mclk_control (struct radeon_device *rdev,bool enable)
The patch just breaks SCLK gating on R6xx again, so no gain here.
Christian.
Am 11.09.2013 10:53, schrieb Markus Trippelsdorf:
Quoted 37 lines · Alex Deucher
Quoted 29 lines
Quoted 22 lines · Eric W. Biederman
Quoted 13 lines · Alex Deucher
Quoted 2 lines · Markus Trippelsdorf
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
They shouldn't be necessary with a proper shutdown, but in this particular case, they are not very expensive. What is expensive is having a separate sanity check functions for all the various hw blocks to teardown everything on startup prior to starting it up in case kexec, etc. left the system in a bad state. It ends up amounting to a full tear down sequence followed by a full start up sequence every time you load the driver.
Quoted 1 line
I can't really comment on the first patch, but the rest seem fine.
Let me reask the question just a little bit.
Is it the sanity checks that are expensive? Or is it the reinitialization that is triggered by the sanity checks that is expensive?
Quoted 5 lines
From what Christian said in the other reply it sounds like this is a
game we will never completely win, but it would be nice to have half a chance in the kexec on panic case to have video. So I am curious to know if the checks are expensive when we are coming at hardware in a clean state.Quoted 7 lines
The particular sanity checks from this patch set are not expensive, but we had previously discussed more extensive sanity checks for other aspects of the chips in prior conversations. Prior to this patch set, the hw is not torn down properly (might have been in the middle of DMA for example) when kexec happens. That's why the sanity checks were added in the first place. With this patch set, the sanity checks shouldn't be necessary.
Quoted 6 lines
I think you're talking past each other.
What Eric worries about is the »kexec on panic« case, where the shutdown method *isn't* called. In this case the sanity checks, that are only superfluous when the hardware was shutdown normally during kexec (the default case), may actually help. And because the checks aren't expensive, it wouldn't hurt to just leave them in place.
When we don't know the state the hardware is in it is really hard to get into a working configuration again, even with the sanity checks in place.
For example you can't just cut power or reset the UVD block without knowing it's state, cause then you have a good chance that you hit into the middle of a register or memory access of the VCPU. This usually results in the next few registers accesses only return either 0x0 or 0xffffffff and after that the system completely locks up.
Isn't it possible to know that we are in a "kexec after panic" case and only try to initialize the display hardware and not all the other blocks? That would at least allow us to get an error message of what happened on the screen and otherwise advise the user to do a clean reset.
Christian.
Quoted 46 lines · Markus Trippelsdorf
Quoted 45 lines · Christian König
Am 09.09.2013 11:21, schrieb Markus Trippelsdorf:
Quoted 24 lines · Eric W. Biederman
Quoted 6 lines
Quoted 4 lines · Markus Trippelsdorf
Here are a couple of patches that get kexec working with radeon devices. I've tested this on my RS780.
Comments or flames are welcome.
Thanks.Quoted 8 lines
A couple of high level comments.
This looks promising for the usual case.
Removing the printk at the end of the kexec path seems a little dubious, what of other cpus, interrupt handlers, etc. Basically estabilishing a new rule on when printk is allowed seems a little dubious at this point, even if it is a useful debugging trick.
OK. I will drop this patch. It doesn't seem to be necessary, because I cannot reproduce the printk related hang anymore.
Quoted 3 lines
Having a clean shutdown of the radeon definitely seems worth doing, because the cases where we care abouty video are when a person is in front of the system.
Yes. But please note that even with
radeon_pci_shutdownimplemented, I still get ring test failures on roughly every eighth kexec boot:Quoted 8 lines
[drm:r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (
0xCAFEDEAD) radeon 0000:01:05.0: disabling GPU accelerationThat's definitely better than the current state of affairs, with ring test failures on every second boot. But I haven't figured out the reason for these failures yet. It's curious that once a ring test failure occurs, it will reliably fail after each kexec invocation, no matter how often repeated. Only a reboot brings the machine back to normal.
The main problem here is that the AMD gfx hardware doesn't really support being reinitialized once booted (for various reasons). It's a (intended) limitation of the hardware design that you can only initialize certain blocks once every power cycle, so the whole approach actually will never work 100% reliable.
All you can hope for is that stopping the hardware while shutting down the old kernel and starting it again results in exactly the same hardware parameters (offsets, clock etc...) otherwise starting the blocks will just fail and you end up with disabled acceleration like above.
Quoted 8 lines
Quoted 1 line
Sorry, but there isn't much we can do about this,
I've tested this further and it turned out that if I revert commit
f5d9b7f0f9on top of my "drm/radeon: Implementradeon_pci_shutdown" patch, the initialization failures seem to go away completely.Any idea what's going on?
You are disabling dynamic power management with that patch reverted. The patch fixed a copy paste typo in the register. Bit 0 (SCLK_PWRMGT_OFF) of register SCLK_PWRMGT_CNTL controls whether dynamic engine clock control is enabled. Bit 0 (GLOBAL_PWRMGT_EN) of register GENERAL_PWRMGT controls whether dynamic power management (dynamic engine/memory/voltage, controls etc.) is enabled at all.
Alex
Quoted 26 lines
Here's the patch:
drivers/gpu/drm/radeon/r600_dpm.c +2
-3@@ -296,9 +296,9 @@ bool r600_dynamicpm_enabled(struct radeon_device *rdev)void r600_enable_sclk_control (struct radeon_device *rdev,bool enable) {if (enable) -WREG32_P (SCLK_PWRMGT_CNTL ,0 , ~SCLK_PWRMGT_OFF ); +WREG32_P (GENERAL_PWRMGT ,0 , ~SCLK_PWRMGT_OFF );else -WREG32_P (SCLK_PWRMGT_CNTL ,SCLK_PWRMGT_OFF , ~SCLK_PWRMGT_OFF ); +WREG32_P (GENERAL_PWRMGT ,SCLK_PWRMGT_OFF , ~SCLK_PWRMGT_OFF ); }void r600_enable_mclk_control (struct radeon_device *rdev,bool enable) --Markus
_______________________________________________
dri-devel mailing list [email protected]
http://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 41 lines · Markus Trippelsdorf
Quoted 32 lines · Alex Deucher
Quoted 24 lines · Eric W. Biederman
Quoted 14 lines · Alex Deucher
Quoted 2 lines · Markus Trippelsdorf
IIRC Alex said the sanity checks are expensive and boot-time could be improved by dropping them. Maybe he can chime in?
They shouldn't be necessary with a proper shutdown, but in this particular case, they are not very expensive. What is expensive is having a separate sanity check functions for all the various hw blocks to teardown everything on startup prior to starting it up in case kexec, etc. left the system in a bad state. It ends up amounting to a full tear down sequence followed by a full start up sequence every time you load the driver.
Quoted 1 line
I can't really comment on the first patch, but the rest seem fine.
Let me reask the question just a little bit.
Is it the sanity checks that are expensive? Or is it the reinitialization that is triggered by the sanity checks that is expensive?
Quoted 5 lines
From what Christian said in the other reply it sounds like this is a game we will never completely win, but it would be nice to have half a chance in the kexec on panic case to have video. So I am curious to know if the checks are expensive when we are coming at hardware in a clean state.
Quoted 7 lines
The particular sanity checks from this patch set are not expensive, but we had previously discussed more extensive sanity checks for other aspects of the chips in prior conversations. Prior to this patch set, the hw is not torn down properly (might have been in the middle of DMA for example) when kexec happens. That's why the sanity checks were added in the first place. With this patch set, the sanity checks shouldn't be necessary.
Quoted 6 lines
I think you're talking past each other.
What Eric worries about is the »kexec on panic« case, where the shutdown method *isn't* called. In this case the sanity checks, that are only superfluous when the hardware was shutdown normally during kexec (the default case), may actually help. And because the checks aren't expensive, it wouldn't hurt to just leave them in place.
For the particular sanity check in this patch set, that's fine. Even with that in place, it only covers a small subset of GPUs and it doesn't cover all possible problematic areas. In general kexec on panic seems like a recipe for trouble. You may end up in the middle of several dma transactions across multiple engines on the GPU. We could add code to tear down every block on the chip on start up in case such an event happened, but like Christian said, some blocks can't really be recovered depending on how things were ended, and then we've doubled the start up time for the driver. Additionally, most of the tear down is dependent on various driver state which we wouldn't have at start up, so most of code would need to re-architected so that it could be called independently of the relevant driver state.
Alex