iommu/arm-smmu: fix "hang" when games exit
Subsystems (2)
PATCH 1/1 iommu/arm-smmu: fix "hang" when games exit
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system is locked up.
A simple solution is to use pm_runtime_put_autosuspend() instead, so we don't immediately suspend the SMMU device.
---
Note: I've tied the autosuspend enable/delay to the consumer device, based on the reasoning that if the consumer device benefits from using an autosuspend delay, then it's corresponding SMMU probably does too. Maybe that is overkill and we should just unconditionally enable autosuspend.
drivers/iommu/arm-smmu.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/iommu/arm-smmu.c +10 -1
@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)
@@ -1445,6 +1445,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec); +#ifdef CONFIG_PM +/* TODO maybe device_link_add() should do this for us? */ +if (dev->power.use_autosuspend) { +pm_runtime_set_autosuspend_delay (smmu->dev, + dev->power.autosuspend_delay); +pm_runtime_use_autosuspend (smmu->dev); + } +#endif + rpm_put:arm_smmu_rpm_put (smmu);return ret;
Signature · 1 line
2.21.0
Robin Murphy
reviewerQuoted 12 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system is locked up.
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.
Quoted 7 lines
Signed-off-by Rob Clark <[email protected]>---
Note: I've tied the autosuspend enable/delay to the consumer device, based on the reasoning that if the consumer device benefits from using an autosuspend delay, then it's corresponding SMMU probably does too. Maybe that is overkill and we should just unconditionally enable autosuspend.
I'm not sure there's really any reason to expect that a supplier's usage model when doing things for itself bears any relation to that of its consumer(s), so I'd certainly lean towards the "unconditional" argument myself.
Of course ideally we'd skip resuming altogether in the map/unmap paths (since resume implies a full TLB reset anyway), but IIRC that approach started to get messy in the context of the initial RPM patchset. I'm planning to fiddle around a bit more to clean up the implementation of the new iommu_flush_ops stuff, so I've made a note to myself to revisit RPM to see if there's a sufficiently clean way to do better. In the meantime, though, I don't have any real objection to using some reasonable autosuspend delay on the principle that if we've been woken up to map/unmap one page, there's a high likelihood that more will follow in short order (and in the configuration slow-paths it won't have much impact either way).
Robin.
Quoted 33 lines
drivers/iommu/arm-smmu.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/iommu/arm-smmu.c +10
-1@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)@@ -1445,6 +1445,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec); +#ifdef CONFIG_PM +/* TODO maybe device_link_add() should do this for us? */ +if (dev->power.use_autosuspend) { +pm_runtime_set_autosuspend_delay (smmu->dev, + dev->power.autosuspend_delay); +pm_runtime_use_autosuspend (smmu->dev); + } +#endif + rpm_put:arm_smmu_rpm_put (smmu);return ret;
Quoted 22 lines · Robin Murphy
Quoted 12 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system is locked up.
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.Quoted 7 lines
Signed-off-by Rob Clark <[email protected]>---
Note: I've tied the autosuspend enable/delay to the consumer device, based on the reasoning that if the consumer device benefits from using an autosuspend delay, then it's corresponding SMMU probably does too. Maybe that is overkill and we should just unconditionally enable autosuspend.
Quoted 4 lines
I'm not sure there's really any reason to expect that a supplier's usage model when doing things for itself bears any relation to that of its consumer(s), so I'd certainly lean towards the "unconditional" argument myself.
Sounds good, I'll respin w/ unconditional autosuspend
Quoted 11 lines
Of course ideally we'd skip resuming altogether in the map/unmap paths (since resume implies a full TLB reset anyway), but IIRC that approach started to get messy in the context of the initial RPM patchset. I'm planning to fiddle around a bit more to clean up the implementation of the new
iommu_flush_opsstuff, so I've made a note to myself to revisit RPM to see if there's a sufficiently clean way to do better. In the meantime, though, I don't have any real objection to using some reasonable autosuspend delay on the principle that if we've been woken up to map/unmap one page, there's a high likelihood that more will follow in short order (and in the configuration slow-paths it won't have much impact either way).
It does sort of remind me about something I was chatting with Jordan the other day.. about how we could possibly skip the TLB inv for unmaps from non-current pagetables once we have per-context pagetables.
The challenge is, since the GPU's command parser is the one switching pagetables, we don't have any race-free way to know which pagetables are current. But we do know which contexts have work queued up for the GPU, so we can know either that a given context definitely isn't current, or that it might be current. And in the "definitely not current" case we could skip TLB inv.
BR,
-R
Quoted 36 lines
Robin.
Quoted 33 lines
drivers/iommu/arm-smmu.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/iommu/arm-smmu.c +10
-1@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)@@ -1445,6 +1445,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec); +#ifdef CONFIG_PM +/* TODO maybe device_link_add() should do this for us? */ +if (dev->power.use_autosuspend) { +pm_runtime_set_autosuspend_delay (smmu->dev, + dev->power.autosuspend_delay); +pm_runtime_use_autosuspend (smmu->dev); + } +#endif + rpm_put:arm_smmu_rpm_put (smmu);return ret;
PATCH v2 1/1 iommu/arm-smmu: fix "hang" when games exit
In reply to Robin MurphyFrom: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system just locked up.
A simple solution is to use pm_runtime_put_autosuspend() instead, so we don't immediately suspend the SMMU device.
---
v1: original
v2: unconditionally use autosuspend, rather than deciding based on what
consumer does
drivers/iommu/arm-smmu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/iommu/arm-smmu.c +4 -1
@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)
@@ -1445,6 +1445,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec); +pm_runtime_set_autosuspend_delay (smmu->dev,20 ); +pm_runtime_use_autosuspend (smmu->dev); + rpm_put:arm_smmu_rpm_put (smmu);return ret;
Signature · 1 line
2.21.0
Quoted 11 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system just locked up.
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.
Quoted 33 lines
Signed-off-by Rob Clark <[email protected]>---
v1: original
v2: unconditionally use autosuspend, rather than deciding based on what
consumer doesdrivers/iommu/arm-smmu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/iommu/arm-smmu.c +4
-1@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)@@ -1445,6 +1445,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec); +pm_runtime_set_autosuspend_delay (smmu->dev,20 ); +pm_runtime_use_autosuspend (smmu->dev); + rpm_put:arm_smmu_rpm_put (smmu);return ret;
Will Deacon
maintainerHi Rob,
Quoted 11 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system just locked up.
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.
Please can you reword the subject to be a bit more useful? The commit message is great, but the subject is a bit like "fix bug in code" to me.
Quoted 23 lines
Signed-off-by Rob Clark <[email protected]>---
v1: original
v2: unconditionally use autosuspend, rather than deciding based on what
consumer doesdrivers/iommu/arm-smmu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/iommu/arm-smmu.c +1
-1@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)
drivers/iommu/arm-smmu.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 1446
Quoted 2 lines
drivers/iommu/arm-smmu.c +0
-0@@ -1445,6 +1445,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec);
Please can you put a comment here explaining what this is doing? An abridged version of the commit message is fine.
drivers/iommu/arm-smmu.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 1449
Quoted 2 lines
drivers/iommu/arm-smmu.c +2
-0+pm_runtime_set_autosuspend_delay (smmu->dev,20 ); +pm_runtime_use_autosuspend (smmu->dev);
Cheers,
Will
Quoted 13 lines · Will Deacon
Hi Rob,
Quoted 9 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system just locked up.
Quoted 5 lines
Quoted 2 lines
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.Please can you reword the subject to be a bit more useful? The commit message is great, but the subject is a bit like "fix bug in code" to me.
yeah, not the best $subject, but I wasn't quite sure how to fit something better in a reasonable # of chars.. maybe something like:
"iommu/arm-smmu: optimize unmap but avoiding toggling runpm state"?
BR,
-R
Quoted 36 lines
Quoted 24 lines
Signed-off-by Rob Clark <[email protected]>---
v1: original
v2: unconditionally use autosuspend, rather than deciding based on what
consumer doesdrivers/iommu/arm-smmu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/iommu/arm-smmu.c +1
-1@@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)static inline void arm_smmu_rpm_put (struct arm_smmu_device *smmu) {if (pm_runtime_enabled (smmu->dev)) -pm_runtime_put (smmu->dev); +pm_runtime_put_autosuspend (smmu->dev); }static struct arm_smmu_domain *to_smmu_domain (struct iommu_domain *dom)drivers/iommu/arm-smmu.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 1446
Quoted 2 lines
drivers/iommu/arm-smmu.c +0
-0@@ -1445,6 +1445,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)/* Looks ok, so add the device to the domain */ ret =arm_smmu_domain_add_master (smmu_domain, fwspec);Please can you put a comment here explaining what this is doing? An abridged version of the commit message is fine.
drivers/iommu/arm-smmu.c static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) · line 1449
Quoted 2 lines
drivers/iommu/arm-smmu.c +2
-0+pm_runtime_set_autosuspend_delay (smmu->dev,20 ); +pm_runtime_use_autosuspend (smmu->dev);Cheers,
Will
Robin Murphy
reviewerQuoted 17 lines · Rob Clark
Quoted 16 lines · Will Deacon
Hi Rob,
Quoted 11 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system just locked up.
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.
Quoted 6 lines
Quoted 2 lines
Please can you reword the subject to be a bit more useful? The commit message is great, but the subject is a bit like "fix bug in code" to me.
yeah, not the best $subject, but I wasn't quite sure how to fit something better in a reasonable # of chars.. maybe something like:
"iommu/arm-smmu: optimize unmap but avoiding toggling runpm state"?
FWIW, I'd be inclined to frame it as something like "avoid pathological RPM behaviour for unmaps".
Robin.
Will Deacon
maintainerQuoted 18 lines · Robin Murphy
Quoted 17 lines · Rob Clark
Quoted 15 lines · Will Deacon
Quoted 11 lines · Rob Clark
From: Rob Clark <[email protected]>
When games, browser, or anything using a lot of GPU buffers exits, there can be many hundreds or thousands of buffers to unmap and free. If the GPU is otherwise suspended, this can cause arm-smmu to resume/suspend for each buffer, resulting 5-10 seconds worth of reprogramming the context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). To the user it would appear that the system just locked up.
A simple solution is to use
pm_runtime_put_autosuspend()instead, so we don't immediately suspend the SMMU device.Please can you reword the subject to be a bit more useful? The commit message is great, but the subject is a bit like "fix bug in code" to me.
Quoted 6 lines
Quoted 3 lines
yeah, not the best $subject, but I wasn't quite sure how to fit something better in a reasonable # of chars.. maybe something like:
"iommu/arm-smmu: optimize unmap but avoiding toggling runpm state"?FWIW, I'd be inclined to frame it as something like "avoid pathological RPM behaviour for unmaps".
LGTM!
Will