SCSI midlayer power management
PATCH 1/1 SCSI midlayer power management
[resend] oops, sorry, forgot to send the patch!
Hi,
This proposed patch implements enough power-management support within the SCSI midlayer to get ACPI S3 working on my system. Changes as follows:
* Add generic_scsi_{suspend,resume} methods to scsi.c
* Add suspend and resume callbacks to the scsi_driver structure, and implement those callbacks in sd.c
* In sd.c, we call sd_shutdown on suspend, in order to synchronize the write-back cache.
* In sd.c, we call sd_rescan from sd_resume in order to ensure that drives have spun up and avoid passing not ready errors back to the block layer.
* In generic_scsi_suspend, we call scsi_device_quiesce before calling the scsi_driver suspend callback. We resume from quiesce state in reverse order in generic_scsi_resume.
ACPI S1 and S4/swsusp are untested, but I think there should be no regressions with S1. To do S1 properly, we probably need to tell the drive to spin down, and I don't know what the SCSI command is for that... For S4, the call to scsi_device_quiesce might pose a problem for the subsequent state dump to disk. But I'm not sure swsusp ever worked for SCSI.
This might help SATA drives, too, but I seem to remember that the SATA layer doesn't properly emulate the SYNCHRONIZE_CACHE command.
Comments, anybody? Can this be applied upstream? I think it's a step in the right direction.
Applies to scsi-misc-2.6
Nathan
===== drivers/scsi/scsi.c 1.145 vs edited =====
drivers/scsi/scsi.c +35 -0
@@ -60,6 +60,7 @@#include <scsi/scsi_cmnd.h> #include <scsi/scsi_dbg.h> #include <scsi/scsi_device.h> +#include <scsi/scsi_driver.h> #include <scsi/scsi_eh.h> #include <scsi/scsi_host.h> #include <scsi/scsi_tcq.h>
@@ -1175,6 +1176,40 @@#define register_scsi_cpu ()#define unregister_scsi_cpu ()#endif /* CONFIG_HOTPLUG_CPU */ + +#ifdef CONFIG_PM +int generic_scsi_suspend (struct device *dev,u32 state) +{ +int err; +struct scsi_device *sdev =to_scsi_device (dev); +struct scsi_driver *drv =to_scsi_driver (dev->driver); + + err =scsi_device_quiesce (sdev); +if (err) +return err; + +if (drv->suspend) +return drv->suspend (dev, state); + +return 0 ; +} + +int generic_scsi_resume (struct device *dev) +{ +int err; +struct scsi_device *sdev =to_scsi_device (dev); +struct scsi_driver *drv =to_scsi_driver (dev->driver); + +if (drv->resume) { + err = drv->resume (dev); +if (err) +return err; + } + +scsi_device_resume (sdev); +return 0 ; +} +#endif /*CONFIG_PM*/ MODULE_DESCRIPTION ("SCSI core" );MODULE_LICENSE ("GPL" );
===== drivers/scsi/scsi_priv.h 1.33 vs edited =====
drivers/scsi/scsi_priv.h +4 -0
@@ -92,6 +92,10 @@static inline void scsi_log_completion (struct scsi_cmnd *cmd,int disposition) { };#endif +#ifdef CONFIG_PM +extern int generic_scsi_suspend (struct device *dev,u32 state); +extern int generic_scsi_resume (struct device *dev); +#endif /* scsi_devinfo.c */ extern int scsi_get_device_flags(struct scsi_device *sdev,
===== drivers/scsi/scsi_sysfs.c 1.52 vs edited =====
drivers/scsi/scsi_sysfs.c +6 -2
@@ -187,8 +187,12 @@ }struct bus_type scsi_bus_type = { - .name ="scsi" , - .match = scsi_bus_match, + .name ="scsi" , + .match = scsi_bus_match, +#ifdef CONFIG_PM + .suspend = generic_scsi_suspend, + .resume = generic_scsi_resume, +#endif };int scsi_sysfs_register (void )
===== drivers/scsi/sd.c 1.154 vs edited =====
drivers/scsi/sd.c +23 -1
@@ -110,6 +110,10 @@static int sd_probe (struct device *);static int sd_remove (struct device *); +#ifdef CONFIG_PM +static int sd_suspend (struct device *,u32 ); +static int sd_resume (struct device *); +#endif static void sd_shutdown (struct device *dev);static void sd_rescan (struct device *);static int sd_init_command (struct scsi_cmnd *);
@@ -126,6 +130,10 @@ }, .rescan = sd_rescan, .init_command = sd_init_command, +#ifdef CONFIG_PM + .suspend = sd_suspend, + .resume = sd_resume, +#endif };/* Device no to disk mapping:
@@ -1549,7 +1557,21 @@scsi_release_request (sreq);printk ("\n" ); -} +} + +#ifdef CONFIG_PM +static int sd_suspend (struct device *dev,u32 state) +{ +sd_shutdown (dev); +return 0 ; +} + +static int sd_resume (struct device *dev) +{ +sd_rescan (dev); +return 0 ; +} +#endif /*CONFIG_PM*/ /** * init_sd - entry point for this driver (both when built in or when
===== include/scsi/scsi_driver.h 1.2 vs edited =====
include/scsi/scsi_driver.h +4 -0
@@ -13,6 +13,10 @@int (*init_command)(struct scsi_cmnd *);void (*rescan)(struct device *); +#ifdef CONFIG_PM +int (*suspend)(struct device *dev,u32 state); +int (*resume)(struct device *dev); +#endif };#define to_scsi_driver (drv) \ container_of((drv), struct scsi_driver, gendrv)
Hi!
Quoted 15 lines
This proposed patch implements enough power-management support within the SCSI midlayer to get ACPI S3 working on my system. Changes as follows:
* Add generic_scsi_{suspend,resume} methods to
scsi.c
* Add suspend and resume callbacks to thescsi_driverstructure, and implement those callbacks insd.c
* Insd.c, we callsd_shutdownon suspend, in order to synchronize the write-back cache.
* Insd.c, we callsd_rescanfromsd_resumein order to ensure that drives have spun up and avoid passing not ready errors back to the block layer.
* Ingeneric_scsi_suspend, we callscsi_device_quiescebefore calling thescsi_driversuspend callback. We resume from quiesce state in reverse order ingeneric_scsi_resume.
Quoted 6 lines
ACPI S1 and S4/swsusp are untested, but I think there should be no regressions with S1. To do S1 properly, we probably need to tell the drive to spin down, and I don't know what the SCSI command is for that... For S4, the call to
scsi_device_quiescemight pose a problem for the subsequent state dump to disk. But I'm not sure swsusp ever worked for SCSI.
swsusp will then resume disk and write the image, that should not be a problem. Is it guaranteed that after generic_scsi_suspend() no DMA is going on?
Anyway, you should try swsusp, preferably on some IDE notebook first and prefereably -mm one, to get feel how it works. It should be possible/easy to make it work with SCSI...
Quoted 5 lines
This might help SATA drives, too, but I seem to remember that the SATA layer doesn't properly emulate the
SYNCHRONIZE_CACHEcommand.Comments, anybody? Can this be applied upstream? I think it's a step in the right direction.
Looks good to me.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 28 lines · Pavel Machek
Hi!
Quoted 15 lines
This proposed patch implements enough power-management support within the SCSI midlayer to get ACPI S3 working on my system. Changes as follows:
* Add generic_scsi_{suspend,resume} methods to
scsi.c
* Add suspend and resume callbacks to thescsi_driverstructure, and implement those callbacks insd.c
* Insd.c, we callsd_shutdownon suspend, in order to synchronize the write-back cache.
* Insd.c, we callsd_rescanfromsd_resumein order to ensure that drives have spun up and avoid passing not ready errors back to the block layer.
* Ingeneric_scsi_suspend, we callscsi_device_quiescebefore calling thescsi_driversuspend callback. We resume from quiesce state in reverse order ingeneric_scsi_resume.Quoted 8 lines
ACPI S1 and S4/swsusp are untested, but I think there should be no regressions with S1. To do S1 properly, we probably need to tell the drive to spin down, and I don't know what the SCSI command is for that... For S4, the call to
scsi_device_quiescemight pose a problem for the subsequent state dump to disk. But I'm not sure swsusp ever worked for SCSI.
Quoted 5 lines
swsusp will then resume disk and write the image, that should not be a problem. Is it guaranteed that after
generic_scsi_suspend()no DMA is going on?
No. Remember that DMA works differently under SCSI than it does under IDE. SCSI DMA is a host controller feature, whereas under IDE it is enabled/disabled at the drive level and the drives have special knowledge of DMA. Since generic_scsi_suspend() is the device level suspend routine, it is called before the host controller's suspend routine, (due to depth first traversal of device tree), which is responsible for disabling the PCI slot. Only after the host controller is suspended will there be no DMA, but if your real question is "can I generically control a SCSI disk with PIO for software suspend" then the answer is NO. For purposes of not suspending the drivers, I haven't looked into how swsusp would see which host adapter owns which drive, but some of the required information seems to be present in sysfs.
Now, I don't know how disabling DMA is supposed to behave under SATA. There may be something extra we have to do that isn't really standard SCSI... SATA is a weird beast and if somebody is using libata it seems it might not be possible to use PIO, since enhanced mode might have to be enabled at the BIOS. But I don't know this stuff, ask jgarzik.
Quoted 5 lines
Anyway, you should try swsusp, preferably on some IDE notebook first and prefereably -mm one, to get feel how it works. It should be possible/easy to make it work with SCSI...
Does it depend on IDE PIO for some reason? Perhaps related to writing out a consistent memory image? I'm not sure if I'm motivated to do anything about that ;)
Quoted 12 lines
Quoted 7 lines
This might help SATA drives, too, but I seem to remember that the SATA layer doesn't properly emulate the
SYNCHRONIZE_CACHEcommand.Comments, anybody? Can this be applied upstream? I think it's a step in the right direction.
Looks good to me.
Pavel
Thanks.
Nathan
Quoted 4 lines · Nathan Bryant
No. Remember that DMA works differently under SCSI than it does under IDE. SCSI DMA is a host controller feature, whereas under IDE it is enabled/disabled at the drive level and the drives have special knowledge of DMA. Since
generic_scsi_suspend()is the device level
Quoted 8 lines
suspend routine, it is called before the host controller's suspend routine, (due to depth first traversal of device tree), which is responsible for disabling the PCI slot. Only after the host controller is suspended will there be no DMA, but if your real question is "can I generically control a SCSI disk with PIO for software suspend" then the answer is NO. For purposes of not suspending the drivers, I haven't looked into how swsusp would see which host adapter owns which drive, but some of the required information seems to be present in sysfs.
Actually, the answer is to most intents and purposes "yes". You are technically correct: there's no way to disable DMA in SCSI. However, once a device is quiesced, it has no outstanding commands, so there will be no outstanding DMA to that device. When all devices on a host have been quiesced, then there will be no DMA at all going on *except* if the user initiates any via another interface (like sending a device probe or doing a unit scan). The guarantee should be strong enough for swsusp to proceed, but we can look at quiescing a host properly (however, we'd need to move to a better host state model than we currently possess).
James
Quoted 4 lines · James Bottomley
Actually, the answer is to most intents and purposes "yes". You are technically correct: there's no way to disable DMA in SCSI. However, once a device is quiesced, it has no outstanding commands, so there will be no outstanding DMA to that device.
In addition we are not doing SCSI target so multi-initiator is ok. One question James - what are the rules for power management with SCSI when we provide termpwr to a shared bus ?
Quoted 3 lines · Alan Cox
In addition we are not doing SCSI target so multi-initiator is ok. One question James - what are the rules for power management with SCSI when we provide termpwr to a shared bus ?
I don't believe the spec addresses that.
The rules we use in HA are that each bus entity supplies termpwr (i.e. all initiators and targets) and each terminator draws its power from the bus). Thus the bus is properly terminated until the last device powers down.
You have to jumper this specially on most SCSI cards (especially if they're at the end of the bus and have internal termination).
James
Quoted 5 lines · James Bottomley
Quoted 3 lines · Alan Cox
In addition we are not doing SCSI target so multi-initiator is ok. One question James - what are the rules for power management with SCSI when we provide termpwr to a shared bus ?
Well there are (at least) two reasons to do shared bus: IP over SCSI and cluster filesystems. At least for cluster filesystems I think the answer is "don't suspend your machine" for the foreseeable future - what happens if your machine goes to sleep while holding cluster FS locks, etc. I doubt any CFS vendors are going to bother to support this any time soon, since it's a HA/server feature and power management is really a desktop/laptop feature.
Quoted 10 lines
Actually, the answer is to most intents and purposes "yes". You are technically correct: there's no way to disable DMA in SCSI. However, once a device is quiesced, it has no outstanding commands, so there will be no outstanding DMA to that device. When all devices on a host have been quiesced, then there will be no DMA at all going on *except* if the user initiates any via another interface (like sending a device probe or doing a unit scan). The guarantee should be strong enough for swsusp to proceed, but we can look at quiescing a host properly (however, we'd need to move to a better host state model than we currently possess).
Some hosts will continuously DMA to memory iirc.. I remember having a problem with 53c8xx on some macs when transitionning from MacOS to Linux because of that.
We need to properly quisce the host, but that's a per host driver thing and shouldn't be too difficult.
Regarding suspend-to-disk, it's fairly easy for the sd driver not to spin down the disk for S4 (only for S3). However, we will still probably do at least a bus reset when waking up...
Pavel: That's one of the reason I wanted an argument to resume() too so drivers can make a difference between the immediate wakeup that happens for writing the image to disk, vs. the real wakeup on resume. In the first case, SCSI can avoid the bus reset, and any kind of re-configuring, in the second case, the full stuff might be necessary.
Ben.
Hi!
Quoted 21 lines
Quoted 9 lines
Actually, the answer is to most intents and purposes "yes". You are technically correct: there's no way to disable DMA in SCSI. However, once a device is quiesced, it has no outstanding commands, so there will be no outstanding DMA to that device. When all devices on a host have been quiesced, then there will be no DMA at all going on *except* if the user initiates any via another interface (like sending a device probe or doing a unit scan). The guarantee should be strong enough for swsusp to proceed, but we can look at quiescing a host properly (however, we'd need to move to a better host state model than we currently possess).
Some hosts will continuously DMA to memory iirc.. I remember having a problem with 53c8xx on some macs when transitionning from MacOS to Linux because of that.
We need to properly quisce the host, but that's a per host driver thing and shouldn't be too difficult.
Regarding suspend-to-disk, it's fairly easy for the sd driver not to spin down the disk for S4 (only for S3). However, we will still probably do at least a bus reset when waking up...
Quoted 5 lines
Pavel: That's one of the reason I wanted an argument to
resume()too so drivers can make a difference between the immediate wakeup that happens for writing the image to disk, vs. the real wakeup on resume. In the first case, SCSI can avoid the bus reset, and any kind of re-configuring, in the second case, the full stuff might be necessary.
Hmm, and it can not be handled by "just remember why you were suspended", because it is one suspend, two resumes...
Yes, I agree that argument will be usefull. Just who does all the driver updating? ;-).
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 6 lines
Hmm, and it can not be handled by "just remember why you were suspended", because it is one suspend, two resumes...
Yes, I agree that argument will be usefull. Just who does all the driver updating? ;-).
At this point, no driver cares, so it's just a matter of fixing the prototypes, I'll leave that to somebody more familiar with sed :)
Ben.
Quoted 3 lines · Benjamin Herrenschmidt
Some hosts will continuously DMA to memory iirc.. I remember having a problem with 53c8xx on some macs when transitionning from MacOS to Linux because of that.
I think you're thinking of the scripts engine? on pre 53c875 chips, yes, this is true. The on-board processor is executing instructions from host memory. However, this is read only in quiescent (waiting for host connect or target reconnect) mode, so shouldn't be a problem for suspend. On the 875 and later, we host the scripts in on-chip memory so they shouldn't be troubling main memory when idling.
Quoted 6 lines
We need to properly quisce the host, but that's a per host driver thing and shouldn't be too difficult.
Regarding suspend-to-disk, it's fairly easy for the sd driver not to spin down the disk for S4 (only for S3). However, we will still probably do at least a bus reset when waking up...
Why? We don't do a bus reset on boot, why should we need to do one on resume? For FC, the equivalent, a LIP Reset can be rather nasty on a SAN and should be avoided.
The slight problem is probably going to be knowing that we may need to spin up devices (for internal ones) before resuming operation.
James
Hi!
Quoted 10 lines
Quoted 3 lines
Some hosts will continuously DMA to memory iirc.. I remember having a problem with 53c8xx on some macs when transitionning from MacOS to Linux because of that.
I think you're thinking of the scripts engine? on pre 53c875 chips, yes, this is true. The on-board processor is executing instructions from host memory. However, this is read only in quiescent (waiting for host connect or target reconnect) mode, so shouldn't be a problem for suspend. On the 875 and later, we host the scripts in on-chip memory so they shouldn't be troubling main memory when idling.
Even read-only access could hurt.... That DMA engine is going to get very unhappy if we change data from under it, right?
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 2 lines · Pavel Machek
Even read-only access could hurt.... That DMA engine is going to get very unhappy if we change data from under it, right?
But we're not planning to change this area of memory (it's a driver allocated coherent mbox) until we power off the box, right? so it should be just like a reboot today.
James
Hi!
Quoted 6 lines
Quoted 2 lines
Even read-only access could hurt.... That DMA engine is going to get very unhappy if we change data from under it, right?
But we're not planning to change this area of memory (it's a driver allocated coherent mbox) until we power off the box, right? so it should be just like a reboot today.
Ok, but what happens on next resume? If coherent mbox is at exactly same place at every boot I guess it could even work, but...
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 2 lines · Pavel Machek
Ok, but what happens on next resume? If coherent mbox is at exactly same place at every boot I guess it could even work, but...
Er, well this is a huge problem then. Even if DMA were stopped, the registers for all these locations need to be altered to change the location of the DMA mboxes. This isn't just a SCSI problem, it's a general device problem (most devices having mboxes programmed by register). If we can't rely on the resuming kernel setting up these registers for us to exactly what they were in the resume image, then we're in a bit of trouble.
Architecturally what you are trying to do is to re POST the SCSI card. Except it's the kernel's job to POST it, so the kernel init code needs to be re-run. I assume that's what the pci suspend/resume calls are supposed to do?
James
Hi!
Quoted 11 lines
Quoted 2 lines
Ok, but what happens on next resume? If coherent mbox is at exactly same place at every boot I guess it could even work, but...
Er, well this is a huge problem then. Even if DMA were stopped, the registers for all these locations need to be altered to change the location of the DMA mboxes. This isn't just a SCSI problem, it's a general device problem (most devices having mboxes programmed by register). If we can't rely on the resuming kernel setting up these registers for us to exactly what they were in the resume image, then we're in a bit of trouble.
Quoted 4 lines
Architecturally what you are trying to do is to re POST the SCSI card. Except it's the kernel's job to POST it, so the kernel init code needs to be re-run. I assume that's what the pci suspend/resume calls are supposed to do?
Yes.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 1 line · Pavel Machek
Yes.
Well, that makes the suspend and resume functions rather complex. They're not going to be coded simply if we have to save and restore the register state of the cards and reinitialise them. I assume if you had to pick three drivers to do this for, that would be aic7xxx, aic79xx and sym_2?
James
Hi!
Quoted 6 lines
Quoted 1 line
Yes.
Well, that makes the suspend and resume functions rather complex. They're not going to be coded simply if we have to save and restore the register state of the cards and reinitialise them. I assume if you had
Can't you simply reuse bootup code? It will no longer be __init, but it should make suspend/resume functions quite simple.
Quoted 2 lines
to pick three drivers to do this for, that would be aic7xxx, aic79xx and
sym_2?
No idea, only SCSI host I owned was some 8-bit isa thing....
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 2 lines · Pavel Machek
Can't you simply reuse bootup code? It will no longer be
__init, but it should make suspend/resume functions quite simple.
Unfortunately, no that simply.
Bootup is all about allocating these areas and initialising the card. Resume will be about initialising the card knowing the existing areas (and the data about the existing areas will have to be part of our persistent data on suspend).
So, modifying the bootup to do something like
if (in_resume)
addr = read from suspend image
else
addr = dma_alloc_coherent(...)
may work.
Quoted 4 lines
Quoted 2 lines
to pick three drivers to do this for, that would be aic7xxx, aic79xx and
sym_2?No idea, only SCSI host I owned was some 8-bit isa thing....
Well, someone who's interested needs to pick a driver. It's usually easier to persuade everyone to add the feature if there's an example to copy...
James
Hi!
Quoted 9 lines
Quoted 2 lines
Can't you simply reuse bootup code? It will no longer be
__init, but it should make suspend/resume functions quite simple.Unfortunately, no that simply.
Bootup is all about allocating these areas and initialising the card. Resume will be about initialising the card knowing the existing areas (and the data about the existing areas will have to be part of our persistent data on suspend).
You can simply free those areas during suspend, so that resume will be same as powerup....
Essentially if you can do insmod and rmmod, you can "simply" emulate rmmod during suspend and emulate "insmod" during resume...
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
to pick three drivers to do this for, that would be aic7xxx, aic79xx and
sym_2?No idea, only SCSI host I owned was some 8-bit isa thing....
Well, someone who's interested needs to pick a driver. It's usually easier to persuade everyone to add the feature if there's an example to copy...
Make it aic7xxx then... Someone already worked on that one.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 8 lines · James Bottomley
Quoted 2 lines · Pavel Machek
Can't you simply reuse bootup code? It will no longer be
__init, but it should make suspend/resume functions quite simple.Unfortunately, no that simply.
Quoted 8 lines
Bootup is all about allocating these areas and initialising the card. Resume will be about initialising the card knowing the existing areas (and the data about the existing areas will have to be part of our
persistent data on suspend).So, modifying the bootup to do something like
if (
in_resume)
addr = read from suspend image
Maybe not that complex. The "suspend image" has become the running kernel by the time we receive the resume request, right? So we maybe just look at the dma base address that we already have in our card-private data structure. If that's all that's needed the S3-resume is more likely to just work for disk suspend.
But in general yes you are right you need to separate the bootup code into "allocate structures" and "program the card for those locations" sections. Personally I think this approach can ultimately be more maintainable than saving and restoring registers.
Quoted 9 lines
else
addr = dma_alloc_coherent(...)may work.
Quoted 3 lines
Quoted 2 lines
to pick three drivers to do this for, that would be aic7xxx, aic79xx and
sym_2?
Quoted 6 lines
Quoted 1 line
No idea, only SCSI host I owned was some 8-bit isa thing....
Well, someone who's interested needs to pick a driver. It's usually easier to persuade everyone to add the feature if there's an example to copy...
Look at my aic7xxx patch. Maybe a little messy but it works. Eh it kind of uses a mix of a save/restore registers technique (inherited from Justin Gibbs' code) and my own "reuse-the-boot-code" approach that I needed to fill in the gaps in his resume code, which was untested
Quoted 5 lines
James
Quoted 9 lines · James Bottomley
Quoted 1 line · Pavel Machek
Yes.
Well, that makes the suspend and resume functions rather complex. They're not going to be coded simply if we have to save and restore the register state of the cards and reinitialise them. I assume if you had
to pick three drivers to do this for, that would be aic7xxx, aic79xx andsym_2?
S3 suspend-to-RAM has the same requirement. When the system enters S3, the PCI slot loses all physical power and the card's registers are completely gone.
So I've already had to fix up the aic7xxx driver to, among other things, reset the DMA base addres on resume. This should help for swsusp. Although maybe there are other requirements for swsusp... I think for swsusp we have to cope with a resume request that seems to come out of nowhere.
Quoted 5 lines
James
Quoted 8 lines · James Bottomley
Quoted 1 line · Pavel Machek
Yes.
Well, that makes the suspend and resume functions rather complex. They're not going to be coded simply if we have to save and restore the register state of the cards and reinitialise them. I assume if you had to pick three drivers to do this for, that would be aic7xxx, aic79xx and
sym_2?
It's not simple for some chips, it's simple for others, in lots of cases, it's just a matter of re-doing the driver init code though.
Ben.
Hi.
Quoted 8 lines · James Bottomley
Er, well this is a huge problem then. Even if DMA were stopped, the registers for all these locations need to be altered to change the location of the DMA mboxes. This isn't just a SCSI problem, it's a general device problem (most devices having mboxes programmed by register). If we can't rely on the resuming kernel setting up these registers for us to exactly what they were in the resume image, then we're in a bit of trouble.
Quoted 4 lines
Architecturally what you are trying to do is to re POST the SCSI card. Except it's the kernel's job to POST it, so the kernel init code needs to be re-run. I assume that's what the pci suspend/resume calls are supposed to do?
I'm not pretending to understand the issues you're talking about, but I do have a question that might possibly be helpful: Pages can be marked with the Nosave flag, so that they're not saved in the image and not overwritten when we copy the old kernel back. Would using Nosave help here at all?
Nigel
Signature · 6 lines
Nigel Cunningham Christian Reformed Church of Tuggeranong PO Box 1004, Tuggeranong, ACT 2901 Many today claim to be tolerant. But true tolerance can cope with others being intolerant.
Howdy again.
Quoted 5 lines · Nigel Cunningham
I'm not pretending to understand the issues you're talking about, but I do have a question that might possibly be helpful: Pages can be marked with the Nosave flag, so that they're not saved in the image and not overwritten when we copy the old kernel back. Would using Nosave help here at all?
Having read the rest of the thread, I can see that's probably not helpful :>
Nigel
Signature · 6 lines
Nigel Cunningham Christian Reformed Church of Tuggeranong PO Box 1004, Tuggeranong, ACT 2901 Many today claim to be tolerant. But true tolerance can cope with others being intolerant.
Quoted 11 lines · James Bottomley
Quoted 2 lines · Pavel Machek
Ok, but what happens on next resume? If coherent mbox is at exactly same place at every boot I guess it could even work, but...
Er, well this is a huge problem then. Even if DMA were stopped, the registers for all these locations need to be altered to change the location of the DMA mboxes. This isn't just a SCSI problem, it's a general device problem (most devices having mboxes programmed by register). If we can't rely on the resuming kernel setting up these registers for us to exactly what they were in the resume image, then we're in a bit of trouble.
Ugh ? What do you mean ? The suspend kernel will snapshot the kernel memory at one point, after the device suspend call has been done. If the device relies on some changes done after that, then it's broken somewhat.
The resume will restore all memory, not MMIO of course, and it's up to the driver to do whatever is necessary to restore operations properly.
Quoted 4 lines
Architecturally what you are trying to do is to re POST the SCSI card. Except it's the kernel's job to POST it, so the kernel init code needs to be re-run. I assume that's what the pci suspend/resume calls are supposed to do?
Yes.
The problem is the same with suspend-to-RAM basically.
Ben.
Quoted 5 lines · James Bottomley
Why? We don't do a bus reset on boot, why should we need to do one on resume? For FC, the equivalent, a LIP Reset can be rather nasty on a SAN and should be avoided.
that can be host specific. aic7xxx does a bus reset on boot, so i preserved this on resume.
don't know why they do it, but they do.
Quoted 4 lines
The slight problem is probably going to be knowing that we may need to spin up devices (for internal ones) before resuming operation.
that's easy for system suspend, but somewhat harder for device suspend
Quoted 9 lines · Nathan Bryant
Quoted 5 lines · James Bottomley
Why? We don't do a bus reset on boot, why should we need to do one on resume? For FC, the equivalent, a LIP Reset can be rather nasty on a SAN and should be avoided.
that can be host specific. aic7xxx does a bus reset on boot, so i preserved this on resume.
It can be changed via the "no_reset" option, that should probably be honored for resume.
Quoted 1 line
don't know why they do it, but they do.
I used to use no_reset, as well as bios no reset (or is it no spin up?) to decrease boot time, until a crash hung the bus and the system could not boot. I suppose letting the bios reset would have gotten around the problem, I can't remember.
-- Patrick Mansfield
Quoted 11 lines · James Bottomley
Quoted 3 lines · Benjamin Herrenschmidt
Some hosts will continuously DMA to memory iirc.. I remember having a problem with 53c8xx on some macs when transitionning from MacOS to Linux because of that.
I think you're thinking of the scripts engine? on pre 53c875 chips, yes, this is true. The on-board processor is executing instructions from host memory. However, this is read only in quiescent (waiting for host connect or target reconnect) mode, so shouldn't be a problem for suspend. On the 875 and later, we host the scripts in on-chip memory so they shouldn't be troubling main memory when idling.
The problem was a 875, the script was writing some kind of status or whatever at regular interval to host memory. Might be specific to the script used by the MacOS driver tho...
Ben.
Hi!
Quoted 6 lines
Quoted 6 lines
swsusp will then resume disk and write the image, that should not be a
problem. Is it guaranteed that aftergeneric_scsi_suspend()no DMA is going on?
Quoted 10 lines
No. Remember that DMA works differently under SCSI than it does under IDE. SCSI DMA is a host controller feature, whereas under IDE it is enabled/disabled at the drive level and the drives have special knowledge of DMA. Since
generic_scsi_suspend()is the device level suspend routine, it is called before the host controller's suspend routine, (due to depth first traversal of device tree), which is responsible for disabling the PCI slot. Only after the host controller is suspended will there be no DMA, but if your real question is "can I generically control a SCSI disk with PIO for software suspend" then the answer is NO. For purposes of not
No, I do not need PIO. I'll probably need host controller support, too, but even w/o it it should work acceptably. Thanks for the answers.
Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms
Quoted 3 lines · Pavel Machek
No, I do not need PIO. I'll probably need host controller support, too, but even w/o it it should work acceptably. Thanks for the answers.
Well, take note of what James wrote. We do guarantee that there will be no further user-initiated disk activity after generic_scsi_suspend. (For the disk but not the rest of the bus.) This doesn't apply to ioctl's/special requests but as James has rightly pointed out this is probably not an issue for realistic intents and purposes. It may actually be an advantage.
Hi.
Quoted 6 lines · Nathan Bryant
Quoted 6 lines
Quoted 6 lines
ACPI S1 and S4/swsusp are untested, but I think there should be no regressions with S1. To do S1 properly, we probably need to tell the drive to spin down, and I don't know what the SCSI command is for that... For S4, the call to
scsi_device_quiescemight pose a problem for the subsequent state dump to disk. But I'm not sure swsusp ever worked for SCSI.
I tried it on an OSDL machine and could suspend (suspend 2), but only resume as far as copying back the original kernel. The problem then looked to me like it was request ids not matching what the drive was expecting (but I'm ignorant of scsi, so might be completely wrong there).
Quoted 3 lines
answer is NO. For purposes of not suspending the drivers, I haven't looked into how swsusp would see which host adapter owns which drive, but some of the required information seems to be present in sysfs.
With my 'device tree' code, I'm getting the struct dev of the device we're using via the struct block_device in the swap_info struct.
Nigel
Signature · 6 lines
Nigel Cunningham Christian Reformed Church of Tuggeranong PO Box 1004, Tuggeranong, ACT 2901 Many today claim to be tolerant. But true tolerance can cope with others being intolerant.
Quoted 14 lines · Nigel Cunningham
Hi.
Quoted 8 lines · Nathan Bryant
Quoted 8 lines
Quoted 8 lines
ACPI S1 and S4/swsusp are untested, but I think there should be no regressions with S1. To do S1 properly, we probably need to tell the drive to spin down, and I don't know what the SCSI command is for that... For S4, the call to
scsi_device_quiescemight pose a problem for the subsequent state dump to disk. But I'm not sure swsusp ever worked for SCSI.
Quoted 7 lines
I tried it on an OSDL machine and could suspend (suspend 2), but only resume as far as copying back the original kernel. The problem then looked to me like it was request ids not matching what the drive was expecting (but I'm ignorant of scsi, so might be completely wrong there).
I saw "no match for command buffer" interrupt storms when I was fixing up aic7xxx for S3. The problem was due to not reprogramming the address of our SCB's on resume. Needed to tell the card the base address for all the DMA structures.
Just to speculate about what would be required for swsusp: you probably need to be using a SCSI LLD that properly implements pci suspend/resume, which implies you need to make sure the card's DMA state machine is flushed and idle before suspend completes. I've got a patch that fixes this much up for aic7xxx. And my other midlayer-level patch may also help... What happens during resume is interesting. I think maybe the problem is not what the drive is expecting, but what the card's state engine is expecting when it tries to map commands to command buffers in DMA space. Maybe you need to suspend the LLD from the context of the kernel that is doing the image load, and then resume from the context of the kernel that was just loaded.
Sounds like this is why Pavel is asking about DMA. So he'll need to manage calling the host adapter's suspend callbacks, not just generic_scsi_suspend. DMA base addresses are likely to change when you load the new kernel image
Quoted 8 lines
Quoted 3 lines
answer is NO. For purposes of not suspending the drivers, I haven't looked into how swsusp would see which host adapter owns which drive, but some of the required information seems to be present in sysfs.
With my 'device tree' code, I'm getting the struct dev of the device we're using via the struct
block_devicein theswap_infostruct.
Right, though you also need to get the host adapter's struct device, if you're not already doing so, that is. Many IDE host drivers don't bother with suspend/resume callbacks at the pci_driver level, but SCSI needs callbacks because the BIOS usually doesn't handle things for us.
Nathan
Hi!
Quoted 12 lines
Quoted 7 lines
I tried it on an OSDL machine and could suspend (suspend 2), but only resume as far as copying back the original kernel. The problem then looked to me like it was request ids not matching what the drive was expecting (but I'm ignorant of scsi, so might be completely wrong there).
I saw "no match for command buffer" interrupt storms when I was fixing up aic7xxx for S3. The problem was due to not reprogramming the address of our SCB's on resume. Needed to tell the card the base address for all the DMA structures.
Quoted 11 lines
Just to speculate about what would be required for swsusp: you probably need to be using a SCSI LLD that properly implements pci suspend/resume, which implies you need to make sure the card's DMA state machine is flushed and idle before suspend completes. I've got a patch that fixes this much up for aic7xxx. And my other midlayer-level patch may also help... What happens during resume is interesting. I think maybe the problem is not what the drive is expecting, but what the card's state engine is expecting when it tries to map commands to command buffers in DMA space. Maybe you need to suspend the LLD from the context of the kernel that is doing the image load, and then resume from the context of the kernel that was just loaded.
Ideally, suspended driver should have no state at all. Like if I send card to suspend with 2.6.8, and when I send it to suspend in 2.6.11, it should be in same state.
Quoted 4 lines
Sounds like this is why Pavel is asking about DMA. So he'll need to manage calling the host adapter's suspend callbacks, not just
generic_scsi_suspend. DMA base addresses are likely to change when you load the new kernel image
sysfs should call host adapter's suspend callbacks... It should work today.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Hi.
Quoted 11 lines · Nathan Bryant
Just to speculate about what would be required for swsusp: you probably need to be using a SCSI LLD that properly implements pci suspend/resume, which implies you need to make sure the card's DMA state machine is flushed and idle before suspend completes. I've got a patch that fixes this much up for aic7xxx. And my other midlayer-level patch may also help... What happens during resume is interesting. I think maybe the problem is not what the drive is expecting, but what the card's state engine is expecting when it tries to map commands to command buffers in DMA space. Maybe you need to suspend the LLD from the context of the kernel that is doing the image load, and then resume from the context of the kernel that was just loaded.
I fully agree. That's what I'm doing at the moment; it's been a while since I looked at swsusp though, so can't say anything about Pavel & Patrick's implementation.
Quoted 7 lines
Quoted 3 lines
With my 'device tree' code, I'm getting the struct dev of the device we're using via the struct
block_devicein theswap_infostruct.Right, though you also need to get the host adapter's struct device, if you're not already doing so, that is. Many IDE host drivers don't bother with suspend/resume callbacks at the
pci_driverlevel, but SCSI needs callbacks because the BIOS usually doesn't handle things for us.
The host adapter isn't in the device's chain of parents?
Nigel
Signature · 6 lines
Nigel Cunningham Christian Reformed Church of Tuggeranong PO Box 1004, Tuggeranong, ACT 2901 Many today claim to be tolerant. But true tolerance can cope with others being intolerant.
Quoted 2 lines · Nigel Cunningham
The host adapter isn't in the device's chain of parents?
It does seem to be, if you look in sysfs. Ok maybe I'm pointing out the obvious ;)