iommu: handle drivers that manage iommu directly
Subsystems (6)
cover iommu: handle drivers that manage iommu directly
From: Rob Clark <[email protected]>
One of the challenges we need to handle to enable the aarch64 laptops upstream is dealing with the fact that the bootloader enables the display and takes the corresponding SMMU context-bank out of BYPASS. Unfortunately, currently, the IOMMU framework attaches a DMA (or potentially an IDENTITY) domain before the driver is probed and has a chance to intervene and shutdown[1] scanout. Which makes things go horribly wrong.
This also happens to solve a problem that is blocking us from supporting per-context pagetables on the GPU, due to domain that is attached before driver has a chance to attach it's own domain for the GPU.
But since the driver is managing it's own iommu domains directly, and does not use dev->iommu_group->default_domain at all, the simple solution to both problems is to just avoid attaching that domain in the first place.
[1] Eventually we want to be able to do a seemless transition from efifb to drm/msm... but first step is to get the core (iommu, clk, genpd) pieces in place, so a first step of disabling the display before first modeset enables us to get all of the dependencies outside of drm/msm in place. And this at least gets us parity with windows (which also appears to do a modeset between bootloader and HLSO). After that there is a bunch of drm/msm work that is probably not interesting to folks outside of dri-devel.
Rob Clark (2):
iommu: add support for drivers that manage iommu explicitly drm/msm: mark devices where iommu is managed by driver
drivers/gpu/drm/msm/adreno/adreno_device.c | 1 + drivers/gpu/drm/msm/disp/dpu1/dpu_kms.c | 1 + drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c | 1 + drivers/gpu/drm/msm/msm_drv.c | 1 + drivers/iommu/iommu.c | 11 +++++++++++ include/linux/device.h | 3 ++- 6 files changed, 17 insertions(+), 1 deletion(-)
Signature · 6 lines
2.20.1 _______________________________________________ Freedreno mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/freedreno
PATCH 2/2 drm/msm: mark devices where iommu is managed by driver
From: Rob Clark <[email protected]>
--- drivers/gpu/drm/msm/adreno/adreno_device.c | 1 + drivers/gpu/drm/msm/disp/dpu1/dpu_kms.c | 1 + drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c | 1 + drivers/gpu/drm/msm/msm_drv.c | 1 + 4 files changed, 4 insertions(+)
drivers/gpu/drm/msm/adreno/adreno_device.c +1 -0
@@ -441,6 +441,7 @@ static struct platform_driver adreno_driver = { .name = "adreno", .of_match_table = dt_match, .pm = &adreno_pm_ops, + .driver_manages_iommu = true, }, };
drivers/gpu/drm/msm/disp/dpu1/dpu_kms.c +1 -0
@@ -1127,6 +1127,7 @@ static struct platform_driver dpu_driver = { .name = "msm_dpu", .of_match_table = dpu_dt_match, .pm = &dpu_pm_ops, + .driver_manages_iommu = true, }, };
drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c +1 -0
@@ -1100,6 +1100,7 @@ static struct platform_driver mdp5_driver = { .name = "msm_mdp", .of_match_table = mdp5_dt_match, .pm = &mdp5_pm_ops, + .driver_manages_iommu = true, }, };
drivers/gpu/drm/msm/msm_drv.c +1 -0
@@ -1352,6 +1352,7 @@ static struct platform_driver msm_platform_driver = { .name = "msm", .of_match_table = dt_match, .pm = &msm_pm_ops, + .driver_manages_iommu = true, }, };
Signature · 6 lines
2.20.1 _______________________________________________ Freedreno mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/freedreno
PATCH 1/2 iommu: add support for drivers that manage iommu explicitly
In reply to Rob ClarkFrom: Rob Clark <[email protected]>
Avoid attaching any non-driver managed domain if the driver indicates that it manages the iommu directly.
This solves a couple problems that drm/msm + arm-smmu has with the iommu framework:
1) In some cases the bootloader takes the iommu out of bypass and enables the display. This is in particular a problem on the aarch64 laptops that exist these days, and modern snapdragon android devices. (Older devices also enabled the display in bootloader but did not take the iommu out of bypass.) Attaching a DMA or IDENTITY domain while scanout is active, before the driver has a chance to intervene, makes things go *boom*
2) We are currently blocked on landing support for GPU per-context pagetables because of the domain attached before driver's ->probe() is called.
This solves both problems.
--- drivers/iommu/iommu.c | 11 +++++++++++ include/linux/device.h | 3 ++- 2 files changed, 13 insertions(+), 1 deletion(-)
drivers/iommu/iommu.c +11 -0
@@ -1573,6 +1573,17 @@ static int __iommu_attach_device(struct iommu_domain *domain, domain->ops->is_attach_deferred(domain, dev))return 0 ; +/* +* If driver is going to manage iommu directly, then avoid +* attaching any non driver managed domain. There could +* be already active dma underway (ie. scanout in case of +* bootloader enabled display), and interfering with that +* will make things go *boom* +*/ +if ((domain->type !=IOMMU_DOMAIN_UNMANAGED ) && + dev->driver && dev->driver->driver_manages_iommu) +return 0 ; +if (unlikely (domain->ops->attach_dev ==NULL ))return -ENODEV ;
include/linux/device.h +2 -1
@@ -282,7 +282,8 @@ struct device_driver {struct module *owner;const char *mod_name;/* used for built-in modules */ -bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */enum probe_type probe_type;const struct of_device_id *of_match_table;
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Robin Murphy
reviewerRe: [PATCH 1/2] iommu: add support for drivers that manage iommu explicitly
Quoted 8 lines · Rob Clark
From: Rob Clark <[email protected]>
Avoid attaching any non-driver managed domain if the driver indicates that it manages the iommu directly.
This solves a couple problems that drm/msm + arm-smmu has with the iommu framework:
Quoted 7 lines
1) In some cases the bootloader takes the iommu out of bypass and enables the display. This is in particular a problem on the aarch64 laptops that exist these days, and modern snapdragon android devices. (Older devices also enabled the display in bootloader but did not take the iommu out of bypass.) Attaching a DMA or IDENTITY domain while scanout is active, before the driver has a chance to intervene, makes things go *boom*
In the general case, we have to assume that things already went boom long ago, as soon as the IOMMU itself was probed and reset. By the time we get to the point of binding of a client driver, also assume that the IOMMU is already powered off and stopping traffic because the RPM device links aren't in place yet and it believes itself unused.
Quoted 3 lines
2) We are currently blocked on landing support for GPU per-context
pagetables because of the domain attached before driver's ->probe() is called.
I'm getting a little fed up of explaining that that problem is specific to the current behaviour of one particular IOMMU driver and trying to work around it anywhere other than in that driver is at best an unreliable hack.
Quoted 1 line
This solves both problems.
For a very, very specific value of "solve"... ;)
Quoted 14 lines
Signed-off-by Rob Clark <[email protected]>--- drivers/iommu/iommu.c | 11 +++++++++++ include/linux/device.h | 3 ++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/iommu/iommu.c +0
-0@@ -1573,6 +1573,17 @@ static int __iommu_attach_device(struct iommu_domain *domain, domain->ops->is_attach_deferred(domain, dev))return 0 ;
drivers/iommu/iommu.c static int __iommu_attach_device(struct iommu_domain *domain, · line 1585
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* If driver is going to manage iommu directly, then avoid +* attaching any non driver managed domain. There could +* be already active dma underway (ie. scanout in case of +* bootloader enabled display), and interfering with that +* will make things go *boom* +*/ +if ((domain->type !=IOMMU_DOMAIN_UNMANAGED ) && + dev->driver && dev->driver->driver_manages_iommu) +return 0 ;
This leaving things half-hanging is really ugly, but more than that it assumes that allocating a default domain in the first place isn't disruptive - I'm not 100% sure that's *always* the case today, and it's definitely likely to change in future as part of improving the current request_dm_for_dev() mechanism. As it happens, those proposed changes would not only break this idea, but make it redundant, since they're about forcing the default domain type to passthrough on a per-device basis, which leads to an equivalent end result to this patch, but in a cleaner and more robust manner.
Robin.
Quoted 19 lines
drivers/iommu/iommu.c +1
-0+if (unlikely (domain->ops->attach_dev ==NULL ))return -ENODEV ;include/linux/device.h +2
-1@@ -282,7 +282,8 @@ struct device_driver {struct module *owner;const char *mod_name;/* used for built-in modules */ -bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */enum probe_type probe_type;const struct of_device_id *of_match_table;
Re: [PATCH 1/2] iommu: add support for drivers that manage iommu explicitly
Quoted 18 lines · Robin Murphy
Quoted 8 lines · Rob Clark
From: Rob Clark <[email protected]>
Avoid attaching any non-driver managed domain if the driver indicates that it manages the iommu directly.
This solves a couple problems that drm/msm + arm-smmu has with the iommu framework:
Quoted 7 lines
1) In some cases the bootloader takes the iommu out of bypass and enables the display. This is in particular a problem on the aarch64 laptops that exist these days, and modern snapdragon android devices. (Older devices also enabled the display in bootloader but did not take the iommu out of bypass.) Attaching a DMA or IDENTITY domain while scanout is active, before the driver has a chance to intervene, makes things go *boom*
Quoted 5 lines
In the general case, we have to assume that things already went boom long ago, as soon as the IOMMU itself was probed and reset. By the time we get to the point of binding of a client driver, also assume that the IOMMU is already powered off and stopping traffic because the RPM device links aren't in place yet and it believes itself unused.
you are correct that this is only part of what is needed to get things working. We also need Bjorn's patch set to inherit SMR and CB config during init:
https://www.spinics.net/lists/arm-kernel/msg732246.html
Quoted 9 lines
Quoted 4 lines
2) We are currently blocked on landing support for GPU per-context
pagetables because of the domain attached before driver's ->probe() is called.I'm getting a little fed up of explaining that that problem is specific to the current behaviour of one particular IOMMU driver and trying to work around it anywhere other than in that driver is at best an unreliable hack.
Perhaps the GPU part of the problem. The display part is not. However I'm fine to move the "don't actually attach" part into arm-smmu if that is preferred. The next person to hit the same problem on a different iommu could certainly move the check or copy it into their iommu driver.
Quoted 3 lines
Quoted 1 line
This solves both problems.
For a very, very specific value of "solve"... ;)
well, "solve" == "it boots fine and doesn't explode"..
I'm certainly happy to entertain alternative suggestions, but these are real problems that need solutions.
BR,
-R
Quoted 56 lines
Quoted 14 lines
Signed-off-by Rob Clark <[email protected]>--- drivers/iommu/iommu.c | 11 +++++++++++ include/linux/device.h | 3 ++- 2 files changed, 13 insertions(+), 1 deletion(-)drivers/iommu/iommu.c +0
-0@@ -1573,6 +1573,17 @@ static int __iommu_attach_device(struct iommu_domain *domain, domain->ops->is_attach_deferred(domain, dev))return 0 ;drivers/iommu/iommu.c static int __iommu_attach_device(struct iommu_domain *domain, · line 1585
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* +* If driver is going to manage iommu directly, then avoid +* attaching any non driver managed domain. There could +* be already active dma underway (ie. scanout in case of +* bootloader enabled display), and interfering with that +* will make things go *boom* +*/ +if ((domain->type !=IOMMU_DOMAIN_UNMANAGED ) && + dev->driver && dev->driver->driver_manages_iommu) +return 0 ;This leaving things half-hanging is really ugly, but more than that it assumes that allocating a default domain in the first place isn't disruptive - I'm not 100% sure that's *always* the case today, and it's definitely likely to change in future as part of improving the current
request_dm_for_dev()mechanism. As it happens, those proposed changes would not only break this idea, but make it redundant, since they're about forcing the default domain type to passthrough on a per-device basis, which leads to an equivalent end result to this patch, but in a cleaner and more robust manner.Robin.
Quoted 19 lines
drivers/iommu/iommu.c +1
-0+if (unlikely (domain->ops->attach_dev ==NULL ))return -ENODEV ;include/linux/device.h +2
-1@@ -282,7 +282,8 @@ struct device_driver {struct module *owner;const char *mod_name;/* used for built-in modules */ -bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */enum probe_type probe_type;const struct of_device_id *of_match_table;
Joerg Roedel
maintainerRe: [PATCH 1/2] iommu: add support for drivers that manage iommu explicitly
In reply to Rob ClarkHi Rob,
Quoted 7 lines · Rob Clark
1) In some cases the bootloader takes the iommu out of bypass and enables the display. This is in particular a problem on the aarch64 laptops that exist these days, and modern snapdragon android devices. (Older devices also enabled the display in bootloader but did not take the iommu out of bypass.) Attaching a DMA or IDENTITY domain while scanout is active, before the driver has a chance to intervene, makes things go *boom*
Just to make sure I get this right: The bootloader inializes the SMMU and creates non-identity mappings for the GPU? And when the SMMU driver in Linux takes over this breaks display output.
drivers/iommu/iommu.c static int __iommu_attach_device(struct iommu_domain *domain, · line 1586
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+/* +* If driver is going to manage iommu directly, then avoid +* attaching any non driver managed domain. There could +* be already active dma underway (ie. scanout in case of +* bootloader enabled display), and interfering with that +* will make things go *boom* +*/ +if ((domain->type !=IOMMU_DOMAIN_UNMANAGED ) && + dev->driver && dev->driver->driver_manages_iommu) +return 0 ; +
When the default domain is attached, there is usually no driver attached yet. I think this needs to be communicated by the firmware to Linux and the code should check against that.
include/linux/device.h struct device_driver { · line 286
Quoted 3 lines
include/linux/device.h +2
-1-bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */
How does this field get set?
Joerg
Re: [PATCH 1/2] iommu: add support for drivers that manage iommu explicitly
Quoted 12 lines · Joerg Roedel
Hi Rob,
Quoted 7 lines · Rob Clark
1) In some cases the bootloader takes the iommu out of bypass and enables the display. This is in particular a problem on the aarch64 laptops that exist these days, and modern snapdragon android devices. (Older devices also enabled the display in bootloader but did not take the iommu out of bypass.) Attaching a DMA or IDENTITY domain while scanout is active, before the driver has a chance to intervene, makes things go *boom*
Quoted 3 lines
Just to make sure I get this right: The bootloader inializes the SMMU and creates non-identity mappings for the GPU? And when the SMMU driver in Linux takes over this breaks display output.
correct
Quoted 12 lines
drivers/iommu/iommu.c static int __iommu_attach_device(struct iommu_domain *domain, · line 1586
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+/* +* If driver is going to manage iommu directly, then avoid +* attaching any non driver managed domain. There could +* be already active dma underway (ie. scanout in case of +* bootloader enabled display), and interfering with that +* will make things go *boom* +*/ +if ((domain->type !=IOMMU_DOMAIN_UNMANAGED ) && + dev->driver && dev->driver->driver_manages_iommu) +return 0 ; +
Quoted 3 lines
When the default domain is attached, there is usually no driver attached yet. I think this needs to be communicated by the firmware to Linux and the code should check against that.
At least for the OF case, it happens in the of_dma_configure() which happens from really_probe(), so there is normally a driver. There are a few exceptional cases, where drivers call of_dma_configure() on their own sub-device without a driver attached (hence the need to check if dev->driver is NULL).
I'm also interested in the ACPI case eventually... the aarch64 "windows" laptops do have ACPI. But for now we are booting with DT since there is quite a lot of work before we get to point of using ACPI. (In particular, under windows, device power management is done thru a Platform Extension Plugin (PEP), but so far linux has no such mechanism.)
We really don't have control of the firmware. But when arm-smmu is probed it can read back the hw state and figure out what is going on (with an RFC series[1] from Bjorn which was posted earlier), so we don't really need to depend on the firmware.
Quoted 5 lines
include/linux/device.h struct device_driver { · line 286
Quoted 3 lines
include/linux/device.h +2
-1-bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */How does this field get set?
It is set in the driver in the second patch[2] in this series.
BR,
-R
[1] https://www.spinics.net/lists/arm-kernel/msg732246.html
[2] https://patchwork.freedesktop.org/patch/315291/
PATCH v2 1/1 iommu: add support for drivers that manage iommu explicitly
In reply to Rob ClarkFrom: Rob Clark <[email protected]>
Avoid attaching any non-driver managed domain if the driver indicates that it manages the iommu directly.
This avoids a problem on devices where the bootloader takes the SMMU out of bypass and enables scanout, such as is the case on snapdragon aarch64 laptops and newer snapdragon android devices. Attaching an IDENTITY or DMA domain before the driver has a chance to intervene will break efifb scanout and start triggering iommu faults.
If the driver manages the iommu directly (as does drm/msm), it can shut down scanout when it is ready to take over the display, before attaching an UNMANAGED domain.
---
v2. Move the check into arm_smmu_attach_dev() (as I *think* this is
what Robin preferred; update commit msg to focus on the display related issue that this solves.
We also need Bjorn's patch set to inherit SMR and CB config during init:
https://www.spinics.net/lists/arm-kernel/msg732246.html
drivers/iommu/arm-smmu.c | 11 +++++++++++ include/linux/device.h | 3 ++- 2 files changed, 13 insertions(+), 1 deletion(-)
drivers/iommu/arm-smmu.c +11 -0
@@ -1411,6 +1411,17 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)return -ENXIO ; } +/* +* If driver is going to manage iommu directly, then avoid +* attaching any non driver managed domain. There could +* be already active dma underway (ie. scanout in case of +* bootloader enabled display), and interfering with that +* will make things go *boom* +*/ +if ((domain->type !=IOMMU_DOMAIN_UNMANAGED ) && + dev->driver && dev->driver->driver_manages_iommu) +return 0 ; +/* * FIXME: The arch/arm DMA API code tries to attach devices to its own * domains between of_xlate() and add_device() - we have no way to cope
include/linux/device.h +2 -1
@@ -282,7 +282,8 @@ struct device_driver {struct module *owner;const char *mod_name;/* used for built-in modules */ -bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */enum probe_type probe_type;const struct of_device_id *of_match_table;
Signature · 1 line
2.20.1
Joerg Roedel
maintainerRe: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
include/linux/device.h struct device_driver { · line 286
Quoted 9 lines · Rob Clark
include/linux/device.h +2
-1@@ -282,7 +282,8 @@ struct device_driver {struct module *owner;const char *mod_name;/* used for built-in modules */ -bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */
Who will set this bit?
Regards,
Joerg
Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 8 lines · Joerg Roedel
Quoted 6 lines · Rob Clark
include/linux/device.h +0
-0@@ -282,7 +282,8 @@ struct device_driver {struct module *owner;const char *mod_name;/* used for built-in modules */
Quoted 6 lines
include/linux/device.h struct device_driver { · line 286
Quoted 3 lines
include/linux/device.h +2
-1-bool suppress_bind_attrs;/* disables bind/unbind via sysfs */ + bool suppress_bind_attrs:1; /* disables bind/unbind via sysfs */ + bool driver_manages_iommu:1; /* driver manages IOMMU explicitly */Who will set this bit?
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
BR,
-R
Joerg Roedel
maintainerRe: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 8 lines · Rob Clark
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
It is more a detail about how the firmware works. IIUC the problem is that the firmware initializes the context mappings for the GPU and the OS doesn't know anything about that and just overwrites them, causing the firmware GPU driver to fail badly.
So I think it is the task of the firmware to tell the OS not to touch the devices mappings until the OS device driver takes over. On x86 there is something similar with the RMRR/unity-map tables from the firmware.
Regards,
Joerg
Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 11 lines · Joerg Roedel
Quoted 8 lines · Rob Clark
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
Quoted 9 lines
It is more a detail about how the firmware works. IIUC the problem is that the firmware initializes the context mappings for the GPU and the OS doesn't know anything about that and just overwrites them, causing the firmware GPU driver to fail badly.
So I think it is the task of the firmware to tell the OS not to touch the devices mappings until the OS device driver takes over. On x86 there is something similar with the RMRR/unity-map tables from the firmware.
Bjorn had a patchset[1] to inherit the config from firmware/bootloader when arm-smmu is probed which handles that part of the problem. My patch is intended to be used on top of his patchset. This seems to me like the best solution, if we don't have control over the firmware.
BR,
-R
Will Deacon
maintainerRe: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 22 lines · Rob Clark
Quoted 11 lines · Joerg Roedel
Quoted 8 lines · Rob Clark
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
Quoted 9 lines
It is more a detail about how the firmware works. IIUC the problem is that the firmware initializes the context mappings for the GPU and the OS doesn't know anything about that and just overwrites them, causing the firmware GPU driver to fail badly.
So I think it is the task of the firmware to tell the OS not to touch the devices mappings until the OS device driver takes over. On x86 there is something similar with the RMRR/unity-map tables from the firmware.
Quoted 4 lines
Bjorn had a patchset[1] to inherit the config from
firmware/bootloaderwhen arm-smmu is probed which handles that part of the problem. My patch is intended to be used on top of his patchset. This seems to me like the best solution, if we don't have control over the firmware.
Hmm, but the feedback from Robin on the thread you cite was that this should be generalised to look more like RMRR, so there seems to be a clear message here.
Will
Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 24 lines · Will Deacon
Quoted 22 lines · Rob Clark
Quoted 20 lines · Joerg Roedel
Quoted 8 lines · Rob Clark
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
It is more a detail about how the firmware works. IIUC the problem is that the firmware initializes the context mappings for the GPU and the OS doesn't know anything about that and just overwrites them, causing the firmware GPU driver to fail badly.
So I think it is the task of the firmware to tell the OS not to touch the devices mappings until the OS device driver takes over. On x86 there is something similar with the RMRR/unity-map tables from the firmware.
Quoted 9 lines
Quoted 4 lines
Bjorn had a patchset[1] to inherit the config from
firmware/bootloaderwhen arm-smmu is probed which handles that part of the problem. My patch is intended to be used on top of his patchset. This seems to me like the best solution, if we don't have control over the firmware.Hmm, but the feedback from Robin on the thread you cite was that this should be generalised to look more like RMRR, so there seems to be a clear message here.
Perhaps it is a lack of creativity, or lack of familiarity w/ iommu vs virtualization, but I'm not quite seeing how RMRR would help.. in particular when dealing with both DT and ACPI cases. So I kinda prefer, when possible, if arm-smmu can figure out what is going on by looking at the hw state at boot (since that approach would work equally well for DT and ACPI).
I *think* (but need to confirm if Bjorn hasn't already) that the memory for the pagetables that firmware/bootloader sets up is already removed from the memory map efi passes to kernel, so we don't need to worry about kernel stomping in-use pagetables.
BR,
-R
Will Deacon
maintainerRe: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 30 lines · Rob Clark
Quoted 29 lines · Will Deacon
Quoted 26 lines · Rob Clark
Quoted 11 lines · Joerg Roedel
Quoted 8 lines · Rob Clark
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
Quoted 9 lines
It is more a detail about how the firmware works. IIUC the problem is that the firmware initializes the context mappings for the GPU and the OS doesn't know anything about that and just overwrites them, causing the firmware GPU driver to fail badly.
So I think it is the task of the firmware to tell the OS not to touch the devices mappings until the OS device driver takes over. On x86 there is something similar with the RMRR/unity-map tables from the firmware.
Bjorn had a patchset[1] to inherit the config from
firmware/bootloaderwhen arm-smmu is probed which handles that part of the problem. My patch is intended to be used on top of his patchset. This seems to me like the best solution, if we don't have control over the firmware.
Quoted 8 lines
Quoted 4 lines
Hmm, but the feedback from Robin on the thread you cite was that this should be generalised to look more like RMRR, so there seems to be a clear message here.
Perhaps it is a lack of creativity, or lack of familiarity w/ iommu vs virtualization, but I'm not quite seeing how RMRR would help.. in particular when dealing with both DT and ACPI cases.
Well, I suppose we'd have something for DT and something for ACPI and we'd try to make them look similar enough that we don't need lots of divergent code in the kernel. The RMRR-like description would describe that, for a particular device, a specific virt:phys mapping needs to exist in the small window between initialising the SMMU and re-initialising the device (GPU).
I would prefer this to be framebuffer-specific, since we're already in flagrant violation of the arm64 boot requirements wrt ongoing DMA and making this too general could lead to all sorts of brain damage. That would probably also allow us to limit the flexibility, by mandating things like alignment and memory attributes.
Having said that, I just realised I'm still a bit confused about the
problem: why does the bootloader require SMMU translation for the GPU at all? If we could limit this whole thing to be identity-mapped/bypassed, then all we need is a per-device flag in the firmware to indicate that the SMMU should be initialised to allow passthrough for transactions originating from that device.
Quoted 3 lines
So I kinda prefer, when possible, if arm-smmu can figure out what is going on by looking at the hw state at boot (since that approach would work equally well for DT and ACPI).
That's not going to fly.
Forcing Linux to infer the state of the system by effectively parsing the hardware configuration directly is fragile, error-prone and may not even be possible in the general case. Worse, if this goes wrong, the symptoms are very likely to be undiagnosable memory corruption, which is pretty awful in my opinion.
Not only would you need separate parsing code for every IOMMU out there, but you'd also need to make Linux aware of device aspects that it otherwise doesn't care about, just in case the firmware decided to use them. Furthermore, running an older kernel on newer hardware (which may have some extensions), could cause the parsing to silently ignore parts of the device that indicate memory regions which are in use. On top of that, there made be device-global state that we are unable to reconfigure and that affect devices other than the ones in question.
So no, I'm very much against this approach and the solution absolutely needs to come in the form of a more abstract description from firmware.
Quoted 4 lines
I *think* (but need to confirm if Bjorn hasn't already) that the memory for the pagetables that
firmware/bootloadersets up is already removed from the memory map efi passes to kernel, so we don't need to worry about kernel stomping in-use pagetables.
It's precisely this sort of fragility that makes me nervous about this whole approach.
Will
Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly
Quoted 41 lines · Will Deacon
Quoted 30 lines · Rob Clark
Quoted 29 lines · Will Deacon
Quoted 26 lines · Rob Clark
Quoted 11 lines · Joerg Roedel
Quoted 8 lines · Rob Clark
It is set by the driver:
https://patchwork.freedesktop.org/patch/315291/
(This doesn't really belong in devicetree, since it isn't a description of the hardware, so the driver is really the only place to set this.. which is fine because it is about a detail of how the driver works.)
Quoted 9 lines
It is more a detail about how the firmware works. IIUC the problem is that the firmware initializes the context mappings for the GPU and the OS doesn't know anything about that and just overwrites them, causing the firmware GPU driver to fail badly.
So I think it is the task of the firmware to tell the OS not to touch the devices mappings until the OS device driver takes over. On x86 there is something similar with the RMRR/unity-map tables from the firmware.
Bjorn had a patchset[1] to inherit the config from
firmware/bootloaderwhen arm-smmu is probed which handles that part of the problem. My patch is intended to be used on top of his patchset. This seems to me like the best solution, if we don't have control over the firmware.Quoted 8 lines
Quoted 4 lines
Hmm, but the feedback from Robin on the thread you cite was that this should be generalised to look more like RMRR, so there seems to be a clear message here.
Perhaps it is a lack of creativity, or lack of familiarity w/ iommu vs virtualization, but I'm not quite seeing how RMRR would help.. in particular when dealing with both DT and ACPI cases.
Quoted 6 lines
Well, I suppose we'd have something for DT and something for ACPI and we'd try to make them look similar enough that we don't need lots of divergent code in the kernel. The RMRR-like description would describe that, for a particular device, a specific virt:phys mapping needs to exist in the small window between initialising the SMMU and re-initialising the device (GPU).
For both DT and ACPI (or perhaps more accurately UEFI and non-UEFI) we often don't have much/any control of the firmware. In the UEFI case, we can at least get the physical address of the scanout buffer from EFI GOP (since VA=PA at that point). In either case we could get the iova by reading back controller state. We kinda just need the iommu driver to not change anything about the context bank the display is using until the display driver is ready to install it's own pagetables.
Initially I just want to shut down display, and then bring it back up w/ my own pagetables.. but eventually, once iommu/clk/genpd issues are sorted upstream, I'm planning to read out more completely the display state, and remap the existing scanout buffer at same iova in my own pagetables/iommu_domain, for a flicker-free display handover... that is a lot more work but at least it is self contained in the display (and bridge/panel) drivers.
Quoted 5 lines
I would prefer this to be framebuffer-specific, since we're already in flagrant violation of the arm64 boot requirements wrt ongoing DMA and making this too general could lead to all sorts of brain damage. That would probably also allow us to limit the flexibility, by mandating things like alignment and memory attributes.
I'd be pretty happy if we could convince qcom to use EFI GOP on android devices too..
Although there is a lot more activity these days with people bringing upstream support to various existing android phones/tablets.. and I'd like to see that continue without downstream hacks due to lack of control over firmware.
Quoted 6 lines
Having said that, I just realised I'm still a bit confused about the
problem: why does the bootloader require SMMU translation for the GPU at all? If we could limit this whole thing to be identity-mapped/bypassed, then all we need is a per-device flag in the firmware to indicate that the SMMU should be initialised to allow passthrough for transactions originating from that device.
I was chatting last night w/ Bjorn on IRC.. and he mentioned that it looked like TTBRn was 0x0. Which I wasn't expecting, and I didn't realize was a legit thing. Maybe the purpose is to allow display to access memory w/ iova==pa but disallow memory access from other devices using different context banks of the same iommu? Maybe this makes more sense to you?
Quoted 11 lines
Quoted 3 lines
So I kinda prefer, when possible, if arm-smmu can figure out what is going on by looking at the hw state at boot (since that approach would work equally well for DT and ACPI).
That's not going to fly.
Forcing Linux to infer the state of the system by effectively parsing the hardware configuration directly is fragile, error-prone and may not even be possible in the general case. Worse, if this goes wrong, the symptoms are very likely to be undiagnosable memory corruption, which is pretty awful in my opinion.
So, I guess in a DT world, we could have some boot-on; flag in the device node.. that tells the iommu driver the stream-id(s) which might be in use. I guess the iommu driver would still need to map that back to context bank, and then leave that CB and stream<->CB mapping alone. But I think that would at least prevent "false positives", which I think is what you are worrying about.
I'm not sure how this would work for ACPI, but tbh I haven't looked yet at how iommu and device are connected w/ ACPI.
Quoted 8 lines
Not only would you need separate parsing code for every IOMMU out there, but you'd also need to make Linux aware of device aspects that it otherwise doesn't care about, just in case the firmware decided to use them. Furthermore, running an older kernel on newer hardware (which may have some extensions), could cause the parsing to silently ignore parts of the device that indicate memory regions which are in use. On top of that, there made be device-global state that we are unable to reconfigure and that affect devices other than the ones in question.
I'm not sure you can avoid having *some* amount of code in each iommu driver, since the exact meaning of iommu-cells can be different.. but I guess we could come up with a generic way of determining which device(s) attached to an iommu might be active when the iommu driver probes.
Quoted 10 lines
So no, I'm very much against this approach and the solution absolutely needs to come in the form of a more abstract description from firmware.
Quoted 4 lines
I *think* (but need to confirm if Bjorn hasn't already) that the memory for the pagetables that
firmware/bootloadersets up is already removed from the memory map efi passes to kernel, so we don't need to worry about kernel stomping in-use pagetables.It's precisely this sort of fragility that makes me nervous about this whole approach.
See above about TTBRn==null.. although if there were actually pagetables, they would need to be reserved memory, otherwise I guess windows would also have a problem w/ stomping pagetables.
BR,
-R