ITS restore/save state when HCC == 0
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Marc Zyngier
maintainerHi Yaohongbo,
In the future, please refrain from sending HTML emails, they don't render very well and force me to reformat your email by hand.
Quoted 7 lines · yaohongbo
Hi, marc.
I met a problem with GIC ITS when I try to power off gic logic in suspend.
In hisilicon hip08, the value of GIC_TYPER.HCC is zero, so that
ITS_FLAGS_SAVE_SUSPEND_STATEwill have no chance to be set to 1.
And that's a good thing. HCC indicates that you have collections that are backed by registers, and not memory. Which means that once the GIC is powered off, the state is lost.
Quoted 2 lines
It goes well for s4, when I simply remove the condition judgement in the code.
What is "s4"? Doing so means you are reprogramming the ITS with mappings that already exist in the tables, and that is UNPRED territory.
<quote>
Behavior is unpredictable if there are interrupts that are mapped to the specified collection and the collection is currently mapped to a Redistributor [...]
</quote>
Quoted 26 lines
---
a/drivers/irqchip/irq-gic-v3-its.c+++
b/drivers/irqchip/irq-gic-v3-its.c(quoted diff) +1
-2@@ -3670,8 +3670,8 @@ static int __init its_probe_one(struct resource *res, ctlr |= GITS_CTLR_ImDe; writel_relaxed(ctlr, its->base + GITS_CTLR); - if (GITS_TYPER_HCC(typer)) - its->flags |= ITS_FLAGS_SAVE_SUSPEND_STATE; + its->flags |= ITS_FLAGS_SAVE_SUSPEND_STATE; err = its_init_domain(handle, its); if (err)@@ -4005,3 +4005,17 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists, return 0;
Quoted 3 lines
}
Do you have any suggestion for this case?
The expectations are that across a GIC power-off, the firmware will restore the state of the GIC (recondiguring the various memory tables), and that this is enough for the ITS to be functional again, having reloaded its state from memory.
Does firmware perform this on your machine? Or are there implementation-specific issues that require the ITS to be reprogrammed?
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 5 lines · Marc Zyngier
Hi Yaohongbo,
In the future, please refrain from sending HTML emails, they don't render very well and force me to reformat your email by hand.
Sorry. I'll pay attention to this next time.
Quoted 13 lines · yaohongbo
Quoted 7 lines
Hi, marc.
I met a problem with GIC ITS when I try to power off gic logic in suspend.
In hisilicon hip08, the value of GIC_TYPER.HCC is zero, so that
ITS_FLAGS_SAVE_SUSPEND_STATEwill have no chance to be set to 1.And that's a good thing. HCC indicates that you have collections that are backed by registers, and not memory. Which means that once the GIC is powered off, the state is lost.
Quoted 5 lines
Quoted 2 lines
It goes well for s4, when I simply remove the condition judgement in the code.
What is "s4"? Doing so means you are reprogramming the ITS with mappings that already exist in the tables, and that is UNPRED territory.
Sorry, I didn't describe it clearly.
S4 means "suspend to disk".
In s4, The its will reinit and malloc an new its address.
My expectation is to reprogram the ITS with original mappings. If ITS_FLAGS_SAVE_SUSPEND_STATE is not set, i'll have no chance to use the original its table mappings.
What should i do if i want to restore its state with hcc == 0?
Thanks,
Hongbo.
Quoted 48 lines
<quote>
Behavior is unpredictable if there are interrupts that are mapped to the specified collection and the collection is currently mapped to a Redistributor [...]
</quote>Quoted 26 lines
---
a/drivers/irqchip/irq-gic-v3-its.c+++
b/drivers/irqchip/irq-gic-v3-its.c(quoted diff) +1
-2@@ -3670,8 +3670,8 @@ static int __init its_probe_one(struct resource *res, ctlr |= GITS_CTLR_ImDe; writel_relaxed(ctlr, its->base + GITS_CTLR); - if (GITS_TYPER_HCC(typer)) - its->flags |= ITS_FLAGS_SAVE_SUSPEND_STATE; + its->flags |= ITS_FLAGS_SAVE_SUSPEND_STATE;err =
its_init_domain(handle, its);if (err)
(quoted diff) +0
-0@@ -4005,3 +4005,17 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists, return 0;Quoted 3 lines
}
Do you have any suggestion for this case?
The expectations are that across a GIC power-off, the firmware will restore the state of the GIC (recondiguring the various memory tables), and that this is enough for the ITS to be functional again, having reloaded its state from memory.
Does firmware perform this on your machine? Or are there implementation-specific issues that require the ITS to be reprogrammed?
Thanks,
M.
Marc Zyngier
maintainer+ James, who wrote most (if not all) of the arm64 hibernate code
Quoted 25 lines · Yao HongBo
Quoted 5 lines · Marc Zyngier
Hi Yaohongbo,
In the future, please refrain from sending HTML emails, they don't render very well and force me to reformat your email by hand.
Sorry. I'll pay attention to this next time.
Quoted 16 lines · yaohongbo
Quoted 7 lines
Hi, marc.
I met a problem with GIC ITS when I try to power off gic logic in suspend.
In hisilicon hip08, the value of GIC_TYPER.HCC is zero, so that
ITS_FLAGS_SAVE_SUSPEND_STATEwill have no chance to be set to 1.And that's a good thing. HCC indicates that you have collections that are backed by registers, and not memory. Which means that once the GIC is powered off, the state is lost.
Quoted 2 lines
It goes well for s4, when I simply remove the condition judgement in the code.
Quoted 6 lines
Quoted 2 lines
What is "s4"? Doing so means you are reprogramming the ITS with mappings that already exist in the tables, and that is UNPRED territory.
Sorry, I didn't describe it clearly.
S4 means "suspend to disk".
In s4, The its will reinit and malloc an new its address.
Huh, hibernate... Yeah, this is not expected to work, I'm afraid.
Quoted 5 lines
My expectation is to reprogram the ITS with original mappings. If
ITS_FLAGS_SAVE_SUSPEND_STATE
is not set, i'll have no chance to use the original its table mappings.What should i do if i want to restore its state with hcc == 0?
HCC is the least of the problems, and there are plenty more issues:
- I'm not sure what guarantees that the tables are at the same address in the booting kernel and the the resumed kernel. That covers all the ITS tables and as well as the RDs'.
- It could well be that restoring the ITS base addresses is enough for everything to resume correctly, but this needs some serious investigation. Worse case, we will need to replay the whole of the ITS programming.
- This is going to interact more or less badly with the normal suspend to RAM code...
- The ITS is only the tip of the iceberg. The whole of the SMMU setup needs to be replayed, or devices won't resume correctly (I just tried on a D05).
Anyway, with the hack below, I've been able to get D05 to resume up to the point where devices try to do DMA, and then it was dead. There is definitely some work to be done there...
M.
drivers/irqchip/irq-gic-v3-its.c +18 -2
@@ -27,6 +27,7 @@#include <linux/of_platform.h> #include <linux/percpu.h> #include <linux/slab.h> +#include <linux/suspend.h> #include <linux/syscore_ops.h> #include <linux/irqchip.h>
@@ -42,6 +43,7 @@#define ITS_FLAGS_WORKAROUND_CAVIUM_22375 (1ULL << 1)#define ITS_FLAGS_WORKAROUND_CAVIUM_23144 (1ULL << 2)#define ITS_FLAGS_SAVE_SUSPEND_STATE (1ULL << 3) +#define ITS_FLAGS_SAVE_HIBERNATE_STATE (1ULL << 4)#define RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING (1 << 0)#define RDIST_FLAGS_RD_TABLES_PREALLOCATED (1 << 1)
@@ -3517,8 +3519,16 @@ static int its_save_disable(void)raw_spin_lock (&its_lock);list_for_each_entry (its, &its_nodes, entry) {void __iomem *base; +u64 flags; -if (!(its->flags &ITS_FLAGS_SAVE_SUSPEND_STATE )) +if (system_entering_hibernation ()) + its->flags |=ITS_FLAGS_SAVE_HIBERNATE_STATE ; + + flags = its->flags; + flags &= (ITS_FLAGS_SAVE_SUSPEND_STATE | +ITS_FLAGS_SAVE_HIBERNATE_STATE ); + +if (!flags)continue ; base = its->base;
@@ -3559,11 +3569,17 @@ static void its_restore_enable(void)raw_spin_lock (&its_lock);list_for_each_entry (its, &its_nodes, entry) {void __iomem *base; +u64 flags;int i; -if (!(its->flags &ITS_FLAGS_SAVE_SUSPEND_STATE )) + flags = its->flags; + flags &= (ITS_FLAGS_SAVE_SUSPEND_STATE | +ITS_FLAGS_SAVE_HIBERNATE_STATE ); +if (!flags)continue ; + its->flags &= ~ITS_FLAGS_SAVE_HIBERNATE_STATE ; + base = its->base;/*
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 121 lines · Marc Zyngier
+ James, who wrote most (if not all) of the arm64 hibernate code
Quoted 28 lines · Yao HongBo
Quoted 5 lines · Marc Zyngier
Hi Yaohongbo,
In the future, please refrain from sending HTML emails, they don't render very well and force me to reformat your email by hand.
Sorry. I'll pay attention to this next time.
Quoted 19 lines · yaohongbo
Quoted 7 lines
Hi, marc.
I met a problem with GIC ITS when I try to power off gic logic in suspend.
In hisilicon hip08, the value of GIC_TYPER.HCC is zero, so that
ITS_FLAGS_SAVE_SUSPEND_STATEwill have no chance to be set to 1.And that's a good thing. HCC indicates that you have collections that
are backed by registers, and not memory. Which means that once the GIC
is powered off, the state is lost.Quoted 3 lines
It goes well for s4, when I simply remove the condition judgement in
the code.Quoted 7 lines
Quoted 3 lines
What is "s4"? Doing so means you are reprogramming the ITS with mappings
that already exist in the tables, and that is UNPRED territory.Sorry, I didn't describe it clearly.
S4 means "suspend to disk".
In s4, The its will reinit and malloc an new its address.Huh, hibernate... Yeah, this is not expected to work, I'm afraid.
Quoted 6 lines
My expectation is to reprogram the ITS with original mappings. If
ITS_FLAGS_SAVE_SUSPEND_STATE
is not set, i'll have no chance to use the original its table mappings.What should i do if i want to restore its state with hcc == 0?
HCC is the least of the problems, and there are plenty more issues:
- I'm not sure what guarantees that the tables are at the same address in the booting kernel and the the resumed kernel. That covers all the ITS tables and as well as the RDs'.
- It could well be that restoring the ITS base addresses is enough for everything to resume correctly, but this needs some serious investigation. Worse case, we will need to replay the whole of the ITS programming.
- This is going to interact more or less badly with the normal suspend to RAM code...
- The ITS is only the tip of the iceberg. The whole of the SMMU setup needs to be replayed, or devices won't resume correctly (I just tried on a D05).
Anyway, with the hack below, I've been able to get D05 to resume up to the point where devices try to do DMA, and then it was dead. There is definitely some work to be done there...
M.drivers/irqchip/irq-gic-v3-its.c +13
-2@@ -27,6 +27,7 @@#include <linux/of_platform.h> #include <linux/percpu.h> #include <linux/slab.h> +#include <linux/suspend.h> #include <linux/syscore_ops.h> #include <linux/irqchip.h> @@ -42,6 +43,7 @@#define ITS_FLAGS_WORKAROUND_CAVIUM_22375 (1ULL << 1)#define ITS_FLAGS_WORKAROUND_CAVIUM_23144 (1ULL << 2)#define ITS_FLAGS_SAVE_SUSPEND_STATE (1ULL << 3) +#define ITS_FLAGS_SAVE_HIBERNATE_STATE (1ULL << 4)#define RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING (1 << 0)#define RDIST_FLAGS_RD_TABLES_PREALLOCATED (1 << 1)@@ -3517,8 +3519,16 @@ static int its_save_disable(void)raw_spin_lock (&its_lock);list_for_each_entry (its, &its_nodes, entry) {void __iomem *base; +u64 flags; -if (!(its->flags &ITS_FLAGS_SAVE_SUSPEND_STATE )) +if (system_entering_hibernation ()) + its->flags |=ITS_FLAGS_SAVE_HIBERNATE_STATE ; + + flags = its->flags; + flags &= (ITS_FLAGS_SAVE_SUSPEND_STATE | +ITS_FLAGS_SAVE_HIBERNATE_STATE ); + +if (!flags)continue ; base = its->base;@@ -3559,11 +3569,17 @@ static void its_restore_enable(void)raw_spin_lock (&its_lock);list_for_each_entry (its, &its_nodes, entry) {void __iomem *base; +u64 flags;int i; -if (!(its->flags &ITS_FLAGS_SAVE_SUSPEND_STATE )) + flags = its->flags;
drivers/irqchip/irq-gic-v3-its.c static void its_restore_enable(void) · line 3585
Quoted 10 lines
drivers/irqchip/irq-gic-v3-its.c +5
-0+ flags &= (ITS_FLAGS_SAVE_SUSPEND_STATE | +ITS_FLAGS_SAVE_HIBERNATE_STATE ); +if (!flags)continue ; + its->flags &= ~ITS_FLAGS_SAVE_HIBERNATE_STATE ; + base = its->base;/*
How about this one to reinit GIC for restore:
- https://source.codeaurora.org/quic/la/kernel/msm-4.14/commit/drivers/irqchip/irq-gic-v3.c?h=msm-4.14&id=b0079fb73c08e195498ba2b2ea9623b0cc0f5fed
Marc Zyngier
maintainerQuoted 128 lines · Ivid Suvarna
Quoted 126 lines · Marc Zyngier
+ James, who wrote most (if not all) of the arm64 hibernate code
Quoted 28 lines · Yao HongBo
Quoted 5 lines · Marc Zyngier
Hi Yaohongbo,
In the future, please refrain from sending HTML emails, they don't render very well and force me to reformat your email by hand.
Sorry. I'll pay attention to this next time.
Quoted 19 lines · yaohongbo
Quoted 7 lines
Hi, marc.
I met a problem with GIC ITS when I try to power off gic logic in suspend.
In hisilicon hip08, the value of GIC_TYPER.HCC is zero, so that
ITS_FLAGS_SAVE_SUSPEND_STATEwill have no chance to be set to 1.And that's a good thing. HCC indicates that you have collections that
are backed by registers, and not memory. Which means that once the GIC
is powered off, the state is lost.Quoted 3 lines
It goes well for s4, when I simply remove the condition judgement in
the code.Quoted 7 lines
Quoted 3 lines
What is "s4"? Doing so means you are reprogramming the ITS with mappings
that already exist in the tables, and that is UNPRED territory.Sorry, I didn't describe it clearly.
S4 means "suspend to disk".
In s4, The its will reinit and malloc an new its address.Huh, hibernate... Yeah, this is not expected to work, I'm afraid.
Quoted 6 lines
My expectation is to reprogram the ITS with original mappings. If
ITS_FLAGS_SAVE_SUSPEND_STATE
is not set, i'll have no chance to use the original its table mappings.What should i do if i want to restore its state with hcc == 0?
HCC is the least of the problems, and there are plenty more issues:
- I'm not sure what guarantees that the tables are at the same address in the booting kernel and the the resumed kernel. That covers all the ITS tables and as well as the RDs'.
- It could well be that restoring the ITS base addresses is enough for everything to resume correctly, but this needs some serious investigation. Worse case, we will need to replay the whole of the ITS programming.
- This is going to interact more or less badly with the normal suspend to RAM code...
- The ITS is only the tip of the iceberg. The whole of the SMMU setup needs to be replayed, or devices won't resume correctly (I just tried on a D05).
Anyway, with the hack below, I've been able to get D05 to resume up to the point where devices try to do DMA, and then it was dead. There is definitely some work to be done there...
M.drivers/irqchip/irq-gic-v3-its.c +16
-2@@ -27,6 +27,7 @@#include <linux/of_platform.h> #include <linux/percpu.h> #include <linux/slab.h> +#include <linux/suspend.h> #include <linux/syscore_ops.h> #include <linux/irqchip.h> @@ -42,6 +43,7 @@#define ITS_FLAGS_WORKAROUND_CAVIUM_22375 (1ULL << 1)#define ITS_FLAGS_WORKAROUND_CAVIUM_23144 (1ULL << 2)#define ITS_FLAGS_SAVE_SUSPEND_STATE (1ULL << 3) +#define ITS_FLAGS_SAVE_HIBERNATE_STATE (1ULL << 4)#define RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING (1 << 0)#define RDIST_FLAGS_RD_TABLES_PREALLOCATED (1 << 1)@@ -3517,8 +3519,16 @@ static int its_save_disable(void)raw_spin_lock (&its_lock);list_for_each_entry (its, &its_nodes, entry) {void __iomem *base; +u64 flags; -if (!(its->flags &ITS_FLAGS_SAVE_SUSPEND_STATE )) +if (system_entering_hibernation ()) + its->flags |=ITS_FLAGS_SAVE_HIBERNATE_STATE ; + + flags = its->flags; + flags &= (ITS_FLAGS_SAVE_SUSPEND_STATE | +ITS_FLAGS_SAVE_HIBERNATE_STATE ); + +if (!flags)continue ; base = its->base;@@ -3559,11 +3569,17 @@ static void its_restore_enable(void)raw_spin_lock (&its_lock);list_for_each_entry (its, &its_nodes, entry) {void __iomem *base; +u64 flags;int i; -if (!(its->flags &ITS_FLAGS_SAVE_SUSPEND_STATE )) + flags = its->flags; + flags &= (ITS_FLAGS_SAVE_SUSPEND_STATE | +ITS_FLAGS_SAVE_HIBERNATE_STATE ); +if (!flags)continue ;
Quoted 8 lines
drivers/irqchip/irq-gic-v3-its.c static void its_restore_enable(void) · line 3585
Quoted 5 lines
drivers/irqchip/irq-gic-v3-its.c +2
-0+ its->flags &= ~ITS_FLAGS_SAVE_HIBERNATE_STATE ; + base = its->base;/* How about this one to reinit GIC for restore:
- https://source.codeaurora.org/quic/la/kernel/msm-4.14/commit/drivers/irqchip/irq-gic-v3.c?h=msm-4.14&id=b0079fb73c08e195498ba2b2ea9623b0cc0f5fed
That's not what we're concerned with at the moment, as there is much more state that is currently missing. Save/restoring registers is the easy part. What needs to be fixed is the way RD memory tables potentially get moved around (and how they can then survive a kexec).
Once we've solved these issues, we'll look at the register state which is likely to already be correct anyway.
M.
Signature · 1 line
Jazz is not dead, it just smells funny.