s390/ism: fix concurrency management in ism_cmd()
Subsystems (4)
Alexandra Winter
maintainerPATCH v2 1/1 s390/ism: fix concurrency management in ism_cmd()
From: Halil Pasic <[email protected]>
The s390x ISM device data sheet clearly states that only one request-response sequence is allowable per ISM function at any point in time. Unfortunately as of today the s390/ism driver in Linux does not honor that requirement. This patch aims to rectify that.
This problem was discovered based on Aliaksei's bug report which states that for certain workloads the ISM functions end up entering error state (with PEC 2 as seen from the logs) after a while and as a consequence connections handled by the respective function break, and for future connection requests the ISM device is not considered -- given it is in a dysfunctional state. During further debugging PEC 3A was observed as well.
A kernel message like
[ 1211.244319] zpci: 061a:00:00.0: Event 0x2 reports an error for PCI function 0x61a is a reliable indicator of the stated function entering error state with PEC 2. Let me also point out that a kernel message like [ 1211.244325] zpci: 061a:00:00.0: The ism driver bound to the device does not support error recovery
is a reliable indicator that the ISM function won't be auto-recovered because the ISM driver currently lacks support for it.
On a technical level, without this synchronization, commands (inputs to the FW) may be partially or fully overwritten (corrupted) by another CPU trying to issue commands on the same function. There is hard evidence that this can lead to DMB token values being used as DMB IOVAs, leading to PEC 2 PCI events indicating invalid DMA. But this is only one of the failure modes imaginable. In theory even completely losing one command and executing another one twice and then trying to interpret the outputs as if the command we intended to execute was actually executed and not the other one is also possible. Frankly, I don't feel confident about providing an exhaustive list of possible consequences.
--- drivers/s390/net/ism_drv.c | 3 +++ include/linux/ism.h | 1 + 2 files changed, 4 insertions(+)
drivers/s390/net/ism_drv.c +3 -0
@@ -130,6 +130,7 @@ static int ism_cmd(struct ism_dev *ism, void *cmd)struct ism_req_hdr *req = cmd;struct ism_resp_hdr *resp = cmd; +spin_lock (&ism->cmd_lock);__ism_write_cmd (ism, req +1 ,sizeof (*req), req->len -sizeof (*req));__ism_write_cmd (ism, req,0 ,sizeof (*req));
@@ -143,6 +144,7 @@ static int ism_cmd(struct ism_dev *ism, void *cmd) }__ism_read_cmd (ism, resp +1 ,sizeof (*resp), resp->len -sizeof (*resp)); out: +spin_unlock (&ism->cmd_lock);return resp->ret; }
@@ -606,6 +608,7 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)return -ENOMEM ;spin_lock_init (&ism->lock); +spin_lock_init (&ism->cmd_lock);dev_set_drvdata (&pdev->dev, ism); ism->pdev = pdev; ism->dev.parent = &pdev->dev;
include/linux/ism.h +1 -0
@@ -28,6 +28,7 @@ struct ism_dmb {struct ism_dev {spinlock_t lock;/* protects the ism device */ +spinlock_t cmd_lock;/* serializes cmds */ struct list_head list;struct pci_dev *pdev;
Signature · 1 line
2.48.1
Quoted 34 lines · Alexandra Winter
From: Halil Pasic <[email protected]>
The s390x ISM device data sheet clearly states that only one request-response sequence is allowable per ISM function at any point in time. Unfortunately as of today the s390/ism driver in Linux does not honor that requirement. This patch aims to rectify that.
This problem was discovered based on Aliaksei's bug report which states that for certain workloads the ISM functions end up entering error state (with PEC 2 as seen from the logs) after a while and as a consequence connections handled by the respective function break, and for future connection requests the ISM device is not considered -- given it is in a dysfunctional state. During further debugging PEC 3A was observed as well.
A kernel message like
[ 1211.244319] zpci: 061a:00:00.0: Event 0x2 reports an error for PCI function 0x61a is a reliable indicator of the stated function entering error state with PEC 2. Let me also point out that a kernel message like [ 1211.244325] zpci: 061a:00:00.0: The ism driver bound to the device does not support error recoveryis a reliable indicator that the ISM function won't be auto-recovered because the ISM driver currently lacks support for it.
On a technical level, without this synchronization, commands (inputs to the FW) may be partially or fully overwritten (corrupted) by another CPU trying to issue commands on the same function. There is hard evidence that this can lead to DMB token values being used as DMB IOVAs, leading to PEC 2 PCI events indicating invalid DMA. But this is only one of the failure modes imaginable. In theory even completely losing one command and executing another one twice and then trying to interpret the outputs as if the command we intended to execute was actually executed and not the other one is also possible. Frankly, I don't feel confident about providing an exhaustive list of possible consequences.
Quoted 7 lines
Fixes 684b89bc39ce ("s390/ism: add device driver for internal shared memory")Reported-by Aliaksei Makarau <[email protected]>Tested-by Mahanta Jambigi <[email protected]>Tested-by Aliaksei Makarau <[email protected]>Signed-off-by Halil Pasic <[email protected]>Reviewed-by Alexandra Winter <[email protected]>Signed-off-by Alexandra Winter <[email protected]>
Thanks for the update.
This patch was applied to netdev/net.git (main) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Paolo Abeni <[email protected]>:
Quoted 8 lines · On Tue, 22 Jul 2025 18:18:17 +0200 you wrote:
From: Halil Pasic <[email protected]>
The s390x ISM device data sheet clearly states that only one request-response sequence is allowable per ISM function at any point in time. Unfortunately as of today the s390/ism driver in Linux does not honor that requirement. This patch aims to rectify that.
[...]
Here is the summary with links:
- [net,v2] s390/ism: fix concurrency management in ism_cmd()
https://git.kernel.org/netdev/net/c/897e8601b9cf
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html