iommu/arm-smmu: Demote error messages to debug in shutdown callback
Subsystems (2)
PATCH 1/1 iommu/arm-smmu: Demote error messages to debug in shutdown callback
Currently on reboot/shutdown, the following messages are displayed on the console as error messages before the system reboots/shutdown.
On SC7180:
arm-smmu 15000000.iommu: removing device with active domains! arm-smmu 5040000.iommu: removing device with active domains!
Demote the log level to debug since it does not offer much help in identifying/fixing any issue as the system is anyways going down and reduce spamming the kernel log.
--- drivers/iommu/arm-smmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/arm-smmu.c +1 -1
@@ -2250,7 +2250,7 @@ static int arm_smmu_device_remove(struct platform_device *pdev)return -ENODEV ;if (!bitmap_empty (smmu->context_map,ARM_SMMU_MAX_CBS )) -dev_err (&pdev->dev,"removing device with active domains!\n" ); +dev_dbg (&pdev->dev,"removing device with active domains!\n" );arm_smmu_bus_init (NULL );iommu_device_unregister (&smmu->iommu);
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Robin Murphy
reviewerQuoted 4 lines · Sai Prakash Ranjan
Currently on reboot/shutdown, the following messages are displayed on the console as error messages before the system reboots/shutdown.
Quoted 8 lines
On SC7180:
arm-smmu 15000000.iommu: removing device with active domains! arm-smmu 5040000.iommu: removing device with active domains!
Demote the log level to debug since it does not offer much help in identifying/fixing any issue as the system is anyways going down and reduce spamming the kernel log.
I've gone back and forth on this pretty much ever since we added the shutdown hook - on the other hand, if any devices *are* still running in those domains at this point, then once we turn off the SMMU and let those IOVAs go out on the bus as physical addresses, all manner of weirdness may ensue. Thus there is an argument for *some* indication that this may happen, although IMO it could be downgraded to at least dev_warn().
Robin.
Quoted 20 lines
Reported-by Douglas Anderson <[email protected]>Signed-off-by Sai Prakash Ranjan <[email protected]>--- drivers/iommu/arm-smmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/iommu/arm-smmu.c +1
-1@@ -2250,7 +2250,7 @@ static int arm_smmu_device_remove(struct platform_device *pdev)return -ENODEV ;if (!bitmap_empty (smmu->context_map,ARM_SMMU_MAX_CBS )) -dev_err (&pdev->dev,"removing device with active domains!\n" ); +dev_dbg (&pdev->dev,"removing device with active domains!\n" );arm_smmu_bus_init (NULL );iommu_device_unregister (&smmu->iommu);
Hi Robin,
Thanks for taking a look at this.
Quoted 14 lines · Robin Murphy
Quoted 4 lines · Sai Prakash Ranjan
Currently on reboot/shutdown, the following messages are displayed on the console as error messages before the system reboots/shutdown.
Quoted 8 lines
On SC7180:
arm-smmu 15000000.iommu: removing device with active domains! arm-smmu 5040000.iommu: removing device with active domains!
Demote the log level to debug since it does not offer much help in identifying/fixing any issue as the system is anyways going down and reduce spamming the kernel log.
Quoted 8 lines
I've gone back and forth on this pretty much ever since we added the shutdown hook - on the other hand, if any devices *are* still running in those domains at this point, then once we turn off the SMMU and let those IOVAs go out on the bus as physical addresses, all manner of weirdness may ensue. Thus there is an argument for *some* indication that this may happen, although IMO it could be downgraded to at least
dev_warn().
Any pointers to the weirdness here after SMMU is turned off? Because if we look at the call sites, device_shutdown is called from kernel_restart_prepare or kernel_shutdown_prepare which would mean system is going down anyways, so do we really care about these error messages or warnings from SMMU?
arm_smmu_device_shutdown
platform_drv_shutdown
device_shutdown
kernel_restart_prepare
kernel_restart
Thanks,
Sai
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 35 lines · Sai Prakash Ranjan
Hi Robin,
Thanks for taking a look at this.
Quoted 14 lines · Robin Murphy
Quoted 4 lines · Sai Prakash Ranjan
Currently on reboot/shutdown, the following messages are displayed on the console as error messages before the system reboots/shutdown.
Quoted 8 lines
On SC7180:
arm-smmu 15000000.iommu: removing device with active domains! arm-smmu 5040000.iommu: removing device with active domains!
Demote the log level to debug since it does not offer much help in identifying/fixing any issue as the system is anyways going down and reduce spamming the kernel log.
Quoted 8 lines
I've gone back and forth on this pretty much ever since we added the shutdown hook - on the other hand, if any devices *are* still running in those domains at this point, then once we turn off the SMMU and let those IOVAs go out on the bus as physical addresses, all manner of weirdness may ensue. Thus there is an argument for *some* indication that this may happen, although IMO it could be downgraded to at least
dev_warn().Any pointers to the weirdness here after SMMU is turned off? Because if we look at the call sites,
device_shutdownis called fromkernel_restart_prepareorkernel_shutdown_preparewhich would mean system is going down anyways, so do we really care about these error messages or warnings from SMMU?
Quoted 6 lines
arm_smmu_device_shutdown
platform_drv_shutdown
device_shutdown
kernel_restart_prepare
kernel_restart
I'd guess that drm/msm is not detaching all of it's UNMANAGED domains in shutdown. Although *presumably* the device_link stuff would prevent the SMMU from shutting down while gpu/display is still active?
If not I think we have bigger problems.
I hadn't really noticed the error msgs before, not sure if that is just because the screen is off by the time they happen or if they are a new warning..
BR,
-R
Hi Rob,
Quoted 44 lines · Rob Clark
Quoted 35 lines · Sai Prakash Ranjan
Hi Robin,
Thanks for taking a look at this.
Quoted 14 lines · Robin Murphy
Quoted 4 lines · Sai Prakash Ranjan
Currently on reboot/shutdown, the following messages are displayed on the console as error messages before the system reboots/shutdown.
Quoted 8 lines
On SC7180:
arm-smmu 15000000.iommu: removing device with active domains! arm-smmu 5040000.iommu: removing device with active domains!
Demote the log level to debug since it does not offer much help in identifying/fixing any issue as the system is anyways going down and reduce spamming the kernel log.
Quoted 8 lines
I've gone back and forth on this pretty much ever since we added the shutdown hook - on the other hand, if any devices *are* still running in those domains at this point, then once we turn off the SMMU and let those IOVAs go out on the bus as physical addresses, all manner of weirdness may ensue. Thus there is an argument for *some* indication that this may happen, although IMO it could be downgraded to at least
dev_warn().Any pointers to the weirdness here after SMMU is turned off? Because if we look at the call sites,
device_shutdownis called fromkernel_restart_prepareorkernel_shutdown_preparewhich would mean system is going down anyways, so do we really care about these error messages or warnings from SMMU?Quoted 6 lines
arm_smmu_device_shutdown
platform_drv_shutdown
device_shutdown
kernel_restart_prepare
kernel_restart
Quoted 5 lines
I'd guess that drm/msm is not detaching all of it's UNMANAGED domains in shutdown. Although *presumably* the
device_linkstuff would prevent the SMMU from shutting down while gpu/display is still active?
If not I think we have bigger problems.
Where is the shutdown callback in drm/msm? I don't see any...
Thanks,
Sai
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Robin Murphy
reviewerQuoted 34 lines · Sai Prakash Ranjan
Hi Robin,
Thanks for taking a look at this.
Quoted 14 lines · Robin Murphy
Quoted 4 lines · Sai Prakash Ranjan
Currently on reboot/shutdown, the following messages are displayed on the console as error messages before the system reboots/shutdown.
Quoted 8 lines
On SC7180:
arm-smmu 15000000.iommu: removing device with active domains! arm-smmu 5040000.iommu: removing device with active domains!
Demote the log level to debug since it does not offer much help in identifying/fixing any issue as the system is anyways going down and reduce spamming the kernel log.
Quoted 8 lines
I've gone back and forth on this pretty much ever since we added the shutdown hook - on the other hand, if any devices *are* still running in those domains at this point, then once we turn off the SMMU and let those IOVAs go out on the bus as physical addresses, all manner of weirdness may ensue. Thus there is an argument for *some* indication that this may happen, although IMO it could be downgraded to at least
dev_warn().Any pointers to the weirdness here after SMMU is turned off? Because if we look at the call sites,
device_shutdownis called fromkernel_restart_prepareorkernel_shutdown_preparewhich would mean system is going down anyways, so do we really care about these error messages or warnings from SMMU?
Quoted 5 lines
arm_smmu_device_shutdown
platform_drv_shutdown
device_shutdown
kernel_restart_prepare
kernel_restart
Imagine your network driver doesn't implement a .shutdown method (so the hardware is still active regardless of device links), happens to have an Rx buffer or descriptor ring DMA-mapped at an IOVA that looks like the physical address of the memory containing some part of the kernel text lower down that call stack, and the MAC receives a broadcast IP packet at about the point arm_smmu_device_shutdown() is returning. Enjoy debugging that ;)
And if coincidental memory corruption seems too far-fetched for your liking, other fun alternatives might include "display tries to scan out from powered-off device, deadlocks interconnect and prevents anything else making progress", or "access to TZC-protected physical address triggers interrupt and over-eager Secure firmware resets system before orderly poweroff has a chance to finish".
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Robin.
Hi Robin,
Quoted 17 lines · Robin Murphy
On 2020-03-27 3:09 pm, Sai Prakash Ranjan wrote:
Imagine your network driver doesn't implement a .shutdown method (so the hardware is still active regardless of device links), happens to have an Rx buffer or descriptor ring DMA-mapped at an IOVA that looks like the physical address of the memory containing some part of the kernel text lower down that call stack, and the MAC receives a broadcast IP packet at about the point
arm_smmu_device_shutdown()is returning. Enjoy debugging that ;)And if coincidental memory corruption seems too far-fetched for your liking, other fun alternatives might include "display tries to scan out from powered-off device, deadlocks interconnect and prevents anything else making progress", or "access to TZC-protected physical address triggers interrupt and over-eager Secure firmware resets system before orderly poweroff has a chance to finish".
Quoted 6 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Thanks for these examples, good to know these scenarios in case we come across these. However, if we see these error/warning messages appear everytime then what will be the credibility of these messages? We will just ignore these messages when these issues you mention actually appears because we see them everytime on reboot or shutdown. So doesn't it make sense to enable these only when we are debugging? We could argue that how will we know the issue could be related to SMMU, but that's the case even now.
The reason why this came up was that, we had a NOC(interconnect) error which does have a logging atleast in QCOM platforms from the secure side(it prints these on the console) after the SMMU err messages and there was a confusion if it was related to these messages. However, NOC error messages did identify the issue with the USB and it was solved later. So these SMMU err/warning messages could be misleading like the above case almost everytime.
The probability of the issues you mentioned occuring is very less than the actual reboot, shutdown scenarios, for ex: we run reboot stress test for thousands of times and these messages don't add anything special in those cases when any issue occurs because they are seen everytime.
Thanks,
Sai
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Hi,
Quoted 8 lines · Sai Prakash Ranjan
Quoted 7 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Quoted 9 lines
Thanks for these examples, good to know these scenarios in case we come across these.
However, if we see these error/warning messages appear everytime then what will be
the credibility of these messages? We will just ignore these messages when
these issues you mention actually appears because we see them everytime on
reboot or shutdown.
I would agree that if these messages are expected to be seen every time, there's no way to fix them, and they're not indicative of any problem then something should be done. Seeing something printed at "dev_error" level with an exclamation point (!) at the end makes me feel like this is something that needs immediate action on my part.
If we really can't do better but feel that the messages need to be there, at least make them dev_info and less scary like:
arm-smmu 15000000.iommu: turning off; DMA should be quiesced before now
...that would still give you a hint in the logs that if you saw a DMA transaction after the message that it was a bug but also wouldn't sound scary to someone who wasn't seeing any other problems.
-Doug
Hi,
Quoted 31 lines · Doug Anderson
Hi,
Quoted 8 lines · Sai Prakash Ranjan
Quoted 7 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Quoted 9 lines
Thanks for these examples, good to know these scenarios in case we come across these.
However, if we see these error/warning messages appear everytime then what will be
the credibility of these messages? We will just ignore these messages when
these issues you mention actually appears because we see them everytime on
reboot or shutdown.I would agree that if these messages are expected to be seen every time, there's no way to fix them, and they're not indicative of any problem then something should be done. Seeing something printed at "
dev_error" level with an exclamation point (!) at the end makes me feel like this is something that needs immediate action on my part.If we really can't do better but feel that the messages need to be there, at least make them
dev_infoand less scary like:
Quoted 7 lines
arm-smmu 15000000.iommu: turning off; DMA should be quiesced before
now...that would still give you a hint in the logs that if you saw a DMA transaction after the message that it was a bug but also wouldn't sound scary to someone who wasn't seeing any other problems.
We can do this if Robin is OK?
-Sai
Signature · 6 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Will Deacon
maintainerQuoted 35 lines · Sai Prakash Ranjan
Quoted 34 lines · Doug Anderson
Quoted 8 lines · Sai Prakash Ranjan
Quoted 7 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Quoted 11 lines
Thanks for these examples, good to know these scenarios in case we come
across these.
However, if we see these error/warning messages appear everytime then what will be
the credibility of these messages? We will just ignore these messages when
these issues you mention actually appears because we see them everytime
on
reboot or shutdown.I would agree that if these messages are expected to be seen every time, there's no way to fix them, and they're not indicative of any problem then something should be done. Seeing something printed at "
dev_error" level with an exclamation point (!) at the end makes me feel like this is something that needs immediate action on my part.If we really can't do better but feel that the messages need to be there, at least make them
dev_infoand less scary like:arm-smmu 15000000.iommu: turning off; DMA should be quiesced before
now
Quoted 6 lines
Quoted 4 lines
...that would still give you a hint in the logs that if you saw a DMA transaction after the message that it was a bug but also wouldn't sound scary to someone who wasn't seeing any other problems.
We can do this if Robin is OK?
It would be nice if you could figure out which domains are still live when the SMMU is being shut down in your case and verify that it *is* infact benign before we start making the message more friendly. As Robin said earlier, rogue DMA is a real nightmare to debug.
Will
Hi Will,
Quoted 41 lines · Will Deacon
Quoted 40 lines · Sai Prakash Ranjan
Quoted 38 lines · Doug Anderson
Quoted 8 lines · Sai Prakash Ranjan
Quoted 7 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Quoted 11 lines
Thanks for these examples, good to know these scenarios in case we come
across these.
However, if we see these error/warning messages appear everytime then what will be
the credibility of these messages? We will just ignore these messages when
these issues you mention actually appears because we see them everytime
on
reboot or shutdown.I would agree that if these messages are expected to be seen every time, there's no way to fix them, and they're not indicative of any problem then something should be done. Seeing something printed at "
dev_error" level with an exclamation point (!) at the end makes me feel like this is something that needs immediate action on my part.If we really can't do better but feel that the messages need to be there, at least make them
dev_infoand less scary like:arm-smmu 15000000.iommu: turning off; DMA should be quiesced before
now...that would still give you a hint in the logs that if you saw a DMA transaction after the message that it was a bug but also wouldn't sound scary to someone who wasn't seeing any other problems.
Quoted 7 lines
Quoted 1 line
We can do this if Robin is OK?
It would be nice if you could figure out which domains are still live when the SMMU is being shut down in your case and verify that it *is* infact benign before we start making the message more friendly. As Robin said earlier, rogue DMA is a real nightmare to debug.
I could see this error message for all the clients of apps_smmu. I checked manually enabling bypass and removing iommus dt property for each client of apps_smmu.
Thanks,
Sai
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Hi,
Quoted 54 lines · Sai Prakash Ranjan
Hi Will,
Quoted 41 lines · Will Deacon
Quoted 40 lines · Sai Prakash Ranjan
Quoted 38 lines · Doug Anderson
Quoted 8 lines · Sai Prakash Ranjan
Quoted 7 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Quoted 11 lines
Thanks for these examples, good to know these scenarios in case we come
across these.
However, if we see these error/warning messages appear everytime then what will be
the credibility of these messages? We will just ignore these messages when
these issues you mention actually appears because we see them everytime
on
reboot or shutdown.I would agree that if these messages are expected to be seen every time, there's no way to fix them, and they're not indicative of any problem then something should be done. Seeing something printed at "
dev_error" level with an exclamation point (!) at the end makes me feel like this is something that needs immediate action on my part.If we really can't do better but feel that the messages need to be there, at least make them
dev_infoand less scary like:arm-smmu 15000000.iommu: turning off; DMA should be quiesced before
now...that would still give you a hint in the logs that if you saw a DMA transaction after the message that it was a bug but also wouldn't sound scary to someone who wasn't seeing any other problems.
Quoted 8 lines
Quoted 1 line
We can do this if Robin is OK?
It would be nice if you could figure out which domains are still live when
the SMMU is being shut down in your case and verify that it *is* infact benign before we start making the message more friendly. As Robin said earlier, rogue DMA is a real nightmare to debug.
Quoted 3 lines
I could see this error message for all the clients of
apps_smmu. I checked manually enabling bypass and removing iommus dt property for each client ofapps_smmu.
Any update on the status here? If I'm reading the conversation above, Robin said: "we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell". Did I understand that properly? If so, it seems like it's fully expected to see this message on every reboot and it doesn't necessarily signify anything bad.
-Doug
Hi,
Quoted 62 lines · Doug Anderson
Hi,
Quoted 54 lines · Sai Prakash Ranjan
Hi Will,
Quoted 41 lines · Will Deacon
Quoted 40 lines · Sai Prakash Ranjan
Quoted 38 lines · Doug Anderson
Quoted 8 lines · Sai Prakash Ranjan
Quoted 7 lines
Of course the fact that in practice we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell, is certainly less than ideal. Like I say, it's not entirely clear-cut either way...
Quoted 11 lines
Thanks for these examples, good to know these scenarios in case we come
across these.
However, if we see these error/warning messages appear everytime then what will be
the credibility of these messages? We will just ignore these messages when
these issues you mention actually appears because we see them everytime
on
reboot or shutdown.I would agree that if these messages are expected to be seen every time, there's no way to fix them, and they're not indicative of any problem then something should be done. Seeing something printed at "
dev_error" level with an exclamation point (!) at the end makes me feel like this is something that needs immediate action on my part.If we really can't do better but feel that the messages need to be there, at least make them
dev_infoand less scary like:arm-smmu 15000000.iommu: turning off; DMA should be quiesced before
now...that would still give you a hint in the logs that if you saw a DMA transaction after the message that it was a bug but also wouldn't sound scary to someone who wasn't seeing any other problems.
Quoted 8 lines
Quoted 1 line
We can do this if Robin is OK?
It would be nice if you could figure out which domains are still live when
the SMMU is being shut down in your case and verify that it *is* infact benign before we start making the message more friendly. As Robin said earlier, rogue DMA is a real nightmare to debug.Quoted 3 lines
I could see this error message for all the clients of
apps_smmu. I checked manually enabling bypass and removing iommus dt property for each client ofapps_smmu.
Quoted 8 lines
Any update on the status here? If I'm reading the conversation above, Robin said: "we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell". Did I understand that properly? If so, it seems like it's fully expected to see this message on every reboot and it doesn't necessarily signify anything bad.
Understanding is the same, waiting for Will and Robin to check if its OK to make the message more friendly.
-Sai
Signature · 6 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Robin Murphy
reviewerOn 2020-04-23 9:17 am, Sai Prakash Ranjan wrote:
[...]
Quoted 11 lines
Quoted 8 lines
Any update on the status here? If I'm reading the conversation above, Robin said: "we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell". Did I understand that properly? If so, it seems like it's fully expected to see this message on every reboot and it doesn't necessarily signify anything bad.
Understanding is the same, waiting for Will and Robin to check if its OK to make the message more friendly.
The way I see it, we essentially just want *something* visible that will correlate with any misbehaviour that *might* result from turning off a possibly-live context. How about simply "disabling translation", at dev_warn or dev_info level?
Robin.
Quoted 11 lines · Robin Murphy
On 2020-04-23 9:17 am, Sai Prakash Ranjan wrote:
[...]Quoted 9 lines
Quoted 8 lines
Any update on the status here? If I'm reading the conversation above, Robin said: "we'll *always* see the warning because there's no way to tear down the default DMA domains, and even if all devices *have* been nicely quiesced there's no way to tell". Did I understand that properly? If so, it seems like it's fully expected to see this message on every reboot and it doesn't necessarily signify anything bad.
Quoted 8 lines
Quoted 2 lines
Understanding is the same, waiting for Will and Robin to check if its OK to make the message more friendly.
The way I see it, we essentially just want *something* visible that will correlate with any misbehaviour that *might* result from turning off a possibly-live context. How about simply "disabling translation", at
dev_warnordev_infolevel?
Sounds good, I'll go with disabling translation with dev_info.
Thanks,
Sai
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation