crypto: cmh - add Rambus CryptoManager Hub core driver and AES
Subsystems (3)
v7 cover crypto: cmh - add Rambus CryptoManager Hub core driver and AES
The Rambus CryptoManager Hub (CMH) is a hardware cryptographic accelerator accessed through a mailbox-based VCQ (Virtual Command Queue) interface. Support for it was first posted as a single series; following maintainer feedback it is now split into smaller, independently reviewable parts, posted least-problematic-first (the per-revision changelog and the prior postings are listed below). This part adds the core platform driver, the shared key/datastore layer and the AES transforms, plus the device-tree binding and the MAINTAINERS entry. AES is the first and least-problematic algorithm family, so the core has a user and the VCQ/DMA/completion data path is reviewable end to end. Later parts add the remaining symmetric ciphers (SM4, ChaCha20-Poly1305) and the hwrng, the hashes and MACs, and finally the asymmetric and post-quantum algorithms together with the /dev/cmh_mgmt management interface.
The core driver binds the device, parses its device-tree description, sets up DMA, the mailbox queue interface and the transaction manager, and exposes sysfs/debugfs instrumentation. The key/datastore patch adds the key provisioning and system-object layer that the AES driver in this series, and the algorithm drivers in the following series, build on.
The transaction manager runs as a kthread that packs VCQ entries, maps DMA, and submits to the least-loaded mailbox; completion is handled by per-mailbox threaded IRQs. The data path is asynchronous: requests return -EINPROGRESS and the driver honors CRYPTO_TFM_REQ_MAY_BACKLOG for queue-full backpressure. The CMH eSW is loaded independently by the hardware before probe; the driver loads no firmware.
The CMH is a platform device matched via device tree (compatible "rambus,cmh-v1030"). Following the inside-secure,eip93 binding model and the DT maintainer's guidance, the schema requires a SoC-specific compatible ahead of the IP fallback and therefore carries no example.
Public hardware documentation:
Product overview: https://www.rambus.com/security/crypto-hub/
No public datasheets or downloadable briefs are currently available. The driver was developed against the Rambus CryptoManager Hub Hardware Reference Manual (Rambus Inc. confidential). Detailed hardware reference is available under NDA from Rambus Inc.; contact the maintainers listed in MAINTAINERS for access during review.
Developed against Herbert Xu's cryptodev tree. checkpatch.pl --strict, sparse (C=2) and a W=1 -Werror build are clean, and dtschema's dt-doc-validate accepts the binding. Exercised on riscv64 and arm64 -- QEMU with the CMH hardware model and a Xilinx VMK180 FPGA with the CMH IP -- under lockdep/PROVE_LOCKING, DEBUG_OBJECTS, KASAN/UBSAN/KMEMLEAK/KFENCE and (arm64) KCSAN with no findings; worst-case stack stays under 1 KB.
A first attempt to post this v7 was partially rejected by the lists' DMARC checks (our outbound mail path altered some messages after they were signed), so the cover letter and part of the series never arrived. This is a complete, clean resend of the same v7 -- no code changes from that attempt.
Changes in v7:
- Corrected the DCO sign-off chain so the co-developer attribution is immediately followed by that author's own sign-off and the submitting author signs last (the DCO point first raised on v3).
- crypto: cmh: close a debugfs force-drain vs teardown use-after-free -- the response-handler lock now also covers freeing the per-mailbox state on teardown, so a force-drain can no longer race it.
- crypto: cmh: reclaim abandoned mailbox scratch with a non-destructive flush ordered in the host command stream instead of a destructive per-mailbox MBX_COMMAND_FLUSH -- removing the cmh_tm_flush_mbx helper and its drain-wait -- so reclaiming scratch no longer cancels a crypto VCQ that shares the mailbox.
- Added the per-revision changelog and the prior-posting links below. The previous posting was sent untagged as "[PATCH 0/5]" and was the sixth revision (v6), so this is v7.
Changes in v6:
- Restructured the single v1-v5 series into smaller, independently reviewable per-maintainer chunks. This first series is the core driver, the shared key/datastore layer and AES; later series add the remaining algorithms and the management ioctl interface.
- dt-bindings: simplified the clock-names schema -- the oneOf collapses to a single items list (core, an optional core-div2 or rt, then rt,
minItems: 1) -- and bounded the per-mailbox queue reg to the hardware range (maximum: 63), following the v5 DT review.
- crypto: cmh: the response handler accumulates the IRQ status bits before clearing the pending flag and releases its mailbox IRQs on teardown even after a suspend; a wedged pinned mailbox now fails with -EIO instead of head-of-line blocking; a second probe is rejected; and debugfs ABORT injection and force-drain take the response-handler lock.
- crypto: cmh: AES-CMAC hardware-accelerates the one-shot .digest() only and forwards the incremental and virtual-address paths to a generic cmac(aes) transform the driver allocates in .init_tfm() (CRYPTO_ALG_NO_FALLBACK + CRYPTO_ALG_REQ_VIRT, as the generic cmac(aes) shash has no export_core and cannot back crypto_ahash_fb()).
- crypto: cmh: moved the datastore-type-to-core-id mapping into the management layer so the key/datastore code no longer depends on the management UAPI header; that header now lands with the ioctl implementation in a later series.
Changes in v5:
- dt-bindings: reworked the compatible and rambus,cores per DT review -- require a SoC-specific compatible ahead of the "rambus,cmh-v1030" IP fallback via the inside-secure,eip93 "not: {}" idiom (no example can validate until a real SoC is upstreamed), switch rambus,cores to a string-array of core names, and replace the positional clocks list with a count-bounded description.
- crypto: cmh: transaction-manager / response-handler race fixes -- FLUSH and drain instead of restarting the eSW into freed slots, serialise the mailbox TAIL doorbell, ABORT a sync timeout only at the mailbox HEAD, and reject an over-large num_vcqs.
Changes in v4:
- dt-bindings: renamed the per-mailbox child to queue@N and described its ring geometry as plain counts (rambus,num-slots / rambus,slot-stride- bytes) rather than log2; renamed the binding to rambus,cmh-v1030.yaml.
- crypto: cmh: convert to module_platform_driver() + dev_err_probe() and drop the single-instance guard; remove informational probe logging; and drop the production module parameters (built-in constants, with debug-only knobs under debugfs).
Changes in v3:
- dt-bindings: reworked the mailbox topology per DT review -- mailbox@N child nodes, with cores discovered from CORE_ENABLE rather than the DT -- and adopted the "rambus" vendor prefix with a device-specific "rambus,cmh-v1030" compatible (the IP revision).
Changes in v2:
- crypto: cmh: gate probe on both eSW readiness bits (MISSION and MISSION2) so an operation offloaded to the sidecar engine cannot race an engine that is not yet up.
The CMH driver was first posted as a single series; following
maintainer feedback it is now split into smaller, independently reviewable parts, numbered continuously across the split. Earlier postings:
v1: https://lore.kernel.org/linux-crypto/[email protected]/
v2: https://lore.kernel.org/linux-crypto/[email protected]/
v3: https://lore.kernel.org/linux-crypto/[email protected]/
v4: https://lore.kernel.org/linux-crypto/[email protected]/
v5: https://lore.kernel.org/linux-crypto/[email protected]/
v6: https://lore.kernel.org/linux-crypto/[email protected]/
---
Alex Ousherovitch (5):
dt-bindings: crypto: add Rambus CryptoManager Hub
crypto: cmh - add core platform driver
crypto: cmh - add key/datastore infrastructure
crypto: cmh - add AES skcipher/aead/cmac
MAINTAINERS: add Rambus CryptoManager Hub (CMH)
Documentation/ABI/testing/debugfs-driver-cmh | 171 ++ Documentation/ABI/testing/sysfs-driver-cmh | 67 + Documentation/crypto/device_drivers/cmh.rst | 340 ++++ Documentation/crypto/device_drivers/index.rst | 1 + .../bindings/crypto/rambus,cmh-v1030.yaml | 144 ++ .../devicetree/bindings/vendor-prefixes.yaml | 2 + MAINTAINERS | 13 + drivers/crypto/Kconfig | 1 + drivers/crypto/Makefile | 1 + drivers/crypto/cmh/Kconfig | 62 + drivers/crypto/cmh/Makefile | 30 + drivers/crypto/cmh/cmh_aes.c | 755 +++++++ drivers/crypto/cmh/cmh_aes_aead.c | 1013 ++++++++++ drivers/crypto/cmh/cmh_aes_cmac.c | 476 +++++ drivers/crypto/cmh/cmh_config.c | 619 ++++++ drivers/crypto/cmh/cmh_debugfs.c | 307 +++ drivers/crypto/cmh/cmh_dma.c | 482 +++++ drivers/crypto/cmh/cmh_key.c | 130 ++ drivers/crypto/cmh/cmh_main.c | 394 ++++ drivers/crypto/cmh/cmh_mqi.c | 344 ++++ drivers/crypto/cmh/cmh_rh.c | 1224 +++++++++++ drivers/crypto/cmh/cmh_sys.c | 376 ++++ drivers/crypto/cmh/cmh_sysfs.c | 108 + drivers/crypto/cmh/cmh_txn.c | 2118 ++++++++++++++++++++ drivers/crypto/cmh/include/cmh.h | 24 + drivers/crypto/cmh/include/cmh_aes.h | 24 + drivers/crypto/cmh/include/cmh_aes_abi.h | 98 + drivers/crypto/cmh/include/cmh_ccp_abi.h | 108 + drivers/crypto/cmh/include/cmh_config.h | 105 + drivers/crypto/cmh/include/cmh_debugfs.h | 93 + drivers/crypto/cmh/include/cmh_dma.h | 249 +++ drivers/crypto/cmh/include/cmh_drbg_abi.h | 67 + drivers/crypto/cmh/include/cmh_eac_abi.h | 44 + drivers/crypto/cmh/include/cmh_hc_abi.h | 162 ++ drivers/crypto/cmh/include/cmh_hcq_abi.h | 222 ++ drivers/crypto/cmh/include/cmh_key.h | 73 + drivers/crypto/cmh/include/cmh_kic_abi.h | 77 + drivers/crypto/cmh/include/cmh_mqi.h | 35 + drivers/crypto/cmh/include/cmh_pke_abi.h | 272 +++ drivers/crypto/cmh/include/cmh_qse_abi.h | 201 ++ drivers/crypto/cmh/include/cmh_registers.h | 161 ++ drivers/crypto/cmh/include/cmh_rh.h | 93 + drivers/crypto/cmh/include/cmh_sm3_abi.h | 79 + drivers/crypto/cmh/include/cmh_sm4_abi.h | 101 + drivers/crypto/cmh/include/cmh_sys.h | 111 + drivers/crypto/cmh/include/cmh_sys_abi.h | 148 ++ drivers/crypto/cmh/include/cmh_sysfs.h | 14 + drivers/crypto/cmh/include/cmh_txn.h | 481 +++++ drivers/crypto/cmh/include/cmh_vcq.h | 288 +++ 49 files changed, 12508 insertions(+)
---
base-commit: 9695df3b1db4ac4534274d38ee16e3b9b0bf910e
change-id: 20261026-cmh-b4-a-v7-be063e120421
Best regards,
Signature · 1 line
Alex Ousherovitch <[email protected]>
PATCH v7 1/5 dt-bindings: crypto: add Rambus CryptoManager Hub
Add device tree binding schema for the Rambus CryptoManager Hub (CMH) hardware crypto accelerator. The binding describes the parent SoC-level node with its SIC register region and one queue@N child node per mailbox the host owns, each carrying a reg (mailbox instance index), an optional interrupt, VCQ ring geometry (rambus,num-slots / rambus,slot-stride-bytes) and a rambus,cores affinity list. Which crypto cores are present is discovered from the SIC CORE_ENABLE register at probe, not described in the device tree.
Register the 'rambus' vendor prefix for Rambus Inc.
--- .../bindings/crypto/rambus,cmh-v1030.yaml | 144 +++++++++++++++++++++ .../devicetree/bindings/vendor-prefixes.yaml | 2 + 2 files changed, 146 insertions(+)
Documentation/devicetree/bindings/crypto/rambus,cmh-v1030.yaml new file +144 -0
@@ -0,0 +1,144 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/crypto/rambus,cmh-v1030.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Rambus CryptoManager Hub (CMH) Hardware Crypto Accelerator + +maintainers: + - Alex Ousherovitch <[email protected]> + - Saravanakrishnan Krishnamoorthy <[email protected]> + - Joel Wittenauer <[email protected]> + +description: | + The Rambus CryptoManager Hub (CMH) is a hardware cryptographic accelerator + accessed via a mailbox-based VCQ (Virtual Command Queue) interface. The + host writes VCQ command sequences into per-mailbox DMA queue buffers and + rings a doorbell; the CMH eSW processes them and signals completion via + interrupt. + + The management host statically partitions the hardware mailboxes across + the SoC's host interfaces at integration time; the set of mailboxes a + given host owns is therefore fixed and not runtime-discoverable (a + mailbox locked to a host reads as unavailable in the SIC availability + register). Each owned mailbox is described by a child node. Which + crypto cores are present is a fixed silicon-build property indicated by + the SIC CORE_ENABLE register, so cores are not described in the device + tree. Clock and reset are owned by the management host; a node + describing a non-management host has no clock or reset provider of its + own, so clocks and reset-gpios are optional. + + CMH gates access to a locked mailbox by a hardware HOST ID presented on + the bus with every access, permitting only the owning host's ID. An + integration must present a single, stable HOST ID for all accesses to a + given mailbox, independent of the issuing CPU (relevant on SMP hosts + whose interconnect encodes the issuing CPU in the HOST ID). + +properties: + compatible: + items: + - not: {} + description: SoC-specific compatible, e.g. vendor,soc-cmh + - const: rambus,cmh-v1030 + + reg: + maxItems: 1 + description: + SIC (System Interface Controller) MMIO region. The registers of + mailbox instance N are at offset N * 0x1000 within this region. + + clocks: + minItems: 1 + maxItems: 3 + description: + The "core" functional clock, and optionally the half-rate "core-div2" + clock (present only on configurations with side-channel-protected + cores) and/or the "rt" real-time tick clock. See clock-names for the + valid combinations. + + clock-names: + minItems: 1 + items: + - const: core + - enum: [core-div2, rt] + - const: rt + + reset-gpios: + maxItems: 1 + description: + Host-controlled reset for the CryptoManager Hub. The hub has two + external, active-low reset inputs -- a power-on reset and a hard + reset; where a board routes one of them to a host GPIO, that line is + described here. + + "#address-cells": + const: 1 + + "#size-cells": + const: 0 + +patternProperties: + "^queue@[0-9a-f]+$": + type: object + description: + One node per hardware mailbox (VCQ command queue) this host owns. + The set of owned mailboxes is fixed by the management host at + integration time and enumerated here. + properties: + reg: + items: + - maximum: 63 + description: + 0-based mailbox instance index (0..63). The instance's + registers are at reg * 0x1000 within the SIC region. + + interrupts: + maxItems: 1 + description: Completion/error interrupt for this mailbox. + + rambus,num-slots: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, + 16384, 32768] + default: 64 + description: + Number of VCQ ring slots for this mailbox's command queue in + host DMA memory. This is a per-board, per-mailbox host-memory + ring geometry -- boards built around the same SoC (hence the + same compatible) may use different ring sizes, so it is + described per mailbox rather than derived from the compatible. + + rambus,slot-stride-bytes: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [128, 256, 512, 1024] + default: 512 + description: + Stride in bytes between consecutive VCQ ring slots for this + mailbox's command queue. Like rambus,num-slots this is a + per-board host-memory ring geometry, not derived from the + compatible. + + rambus,cores: + $ref: /schemas/types.yaml#/definitions/string-array + items: + enum: [hc, aes, sm4, sm3, hcq, qse, pke, ccp] + description: | + Core-affinity list: the crypto cores whose work is dispatched to + this mailbox. A core may appear on at most one mailbox. Cores + not listed on any mailbox are load-balanced across all mailboxes. + Optional (default: none -- the mailbox only serves the + load-balanced pool). + + required: + - reg + + additionalProperties: false + +required: + - compatible + - reg + - "#address-cells" + - "#size-cells" + +additionalProperties: false
Documentation/devicetree/bindings/vendor-prefixes.yaml +2 -0
@@ -1397,6 +1397,8 @@ patternProperties: description: RaidSonic Technology GmbH "^ralink,.*": description: Mediatek/Ralink Technology Corp. + "^rambus,.*": + description: Rambus Inc. "^ramtron,.*": description: Ramtron International "^raspberrypi,.*":
Signature · 1 line
2.43.7
Re: [PATCH v7 1/5] dt-bindings: crypto: add Rambus CryptoManager Hub
Quoted 11 lines · On 26/10/2026 20:44, Alex Ousherovitch via B4 Relay wrote:
From: Alex Ousherovitch <[email protected]>
Add device tree binding schema for the Rambus CryptoManager Hub (CMH) hardware crypto accelerator. The binding describes the parent SoC-level node with its SIC register region and one queue@N child node per mailbox the host owns, each carrying a reg (mailbox instance index), an optional interrupt, VCQ ring geometry (rambus,num-slots / rambus,slot-stride-bytes) and a rambus,cores affinity list. Which crypto cores are present is discovered from the SIC
CORE_ENABLEregister at probe, not described in the device tree.
Quoted 5 lines
Register the 'rambus' vendor prefix for Rambus Inc.
Co-developed-by Saravanakrishnan Krishnamoorthy <[email protected]>Signed-off-by Saravanakrishnan Krishnamoorthy <[email protected]>Signed-off-by Alex Ousherovitch <[email protected]>
Why is this sent multiple times?
b4 diff '<[email protected]>' Looking up [email protected] Checking for older revisions
Grabbing search results from lore.kernel.org
---
Analyzing 8 messages in the thread
Could not find lower series to compare against.
And both variants are present in Patchwork, so not sure what you want us to do here.
Best regards,
Krzysztof
RE: [PATCH v7 1/5] dt-bindings: crypto: add Rambus CryptoManager Hub
Quoted 1 line · Krzysztof Kozlowski
Why is this sent multiple times?
...
Quoted 2 lines
And both variants are present in Patchwork, so not sure what you want us to do here.
Apologies for the duplicate - it's a mail-delivery artifact, not two submissions.
An initial send of this v7 (23 Oct) went out through our corporate SMTP, which the lists' DMARC check partially rejected: the outbound filter altered some messages after they were DKIM-signed, so the cover letter and the MAINTAINERS patch were dropped while patches 1-4 still reached the list. I resent the identical v7 (26 Oct) through the b4/kernel.org relay - this thread - with the full cover letter, all five patches and a base-commit. No code changed between the two.
So the authoritative series is the 26 Oct one ([email protected]). I've marked the 23 Oct copies (patches 1-4) Superseded in Patchwork, so only the 26 Oct series remains active. Future rerolls will go out only via the b4 relay, so this won't recur.
Sorry for the noise.
Alex
PATCH v7 2/5 crypto: cmh - add core platform driver
In reply to Alex OusherovitchAdd the Rambus CryptoManager Hub (CMH) hardware crypto accelerator core platform driver. This patch provides:
- Platform driver registration and probe/remove lifecycle
- Hardware configuration and core discovery
- Mailbox Queue Interface (MQI) for VCQ command submission
- Transaction manager with async completion and backlog support
- Result handler with threaded-IRQ completion
- DMA buffer management
- Debugfs instrumentation (when CONFIG_CRYPTO_DEV_CMH_DEBUG=y)
- Sysfs attributes (fw_version, hw_version, boot_status, mbx_available, mbx_count)
- Kconfig and Makefile integration
No crypto algorithms are registered yet -- those follow in subsequent patches.
The driver communicates with the hardware via a mailbox-based VCQ (Virtual Command Queue) interface. Each crypto operation is packed into VCQ command entries, submitted to a mailbox, and completed asynchronously via interrupt.
MODULE_IMPORT_NS(CRYPTO_INTERNAL) imports two symbols:
- crypto_cipher_setkey() [crypto/cipher.c, EXPORT_SYMBOL_NS_GPL]
- crypto_cipher_encrypt_one() [crypto/cipher.c, EXPORT_SYMBOL_NS_GPL]
These are the single-block cipher API used for the CCM empty-input tag computation software fallback (2 ECB encryptions + XOR). No public wrapper exists; this is the same pattern used by in-tree crypto/ccm.c.
--- Documentation/ABI/testing/debugfs-driver-cmh | 171 ++ Documentation/ABI/testing/sysfs-driver-cmh | 67 + Documentation/crypto/device_drivers/cmh.rst | 340 ++++ Documentation/crypto/device_drivers/index.rst | 1 + drivers/crypto/Kconfig | 1 + drivers/crypto/Makefile | 1 + drivers/crypto/cmh/Kconfig | 62 + drivers/crypto/cmh/Makefile | 25 + drivers/crypto/cmh/cmh_config.c | 619 ++++++++ drivers/crypto/cmh/cmh_debugfs.c | 307 ++++ drivers/crypto/cmh/cmh_dma.c | 482 ++++++ drivers/crypto/cmh/cmh_main.c | 369 +++++ drivers/crypto/cmh/cmh_mqi.c | 344 ++++ drivers/crypto/cmh/cmh_rh.c | 1224 ++++++++++++++ drivers/crypto/cmh/cmh_sysfs.c | 108 ++ drivers/crypto/cmh/cmh_txn.c | 2118 +++++++++++++++++++++++++ drivers/crypto/cmh/include/cmh.h | 24 + drivers/crypto/cmh/include/cmh_aes_abi.h | 98 ++ drivers/crypto/cmh/include/cmh_ccp_abi.h | 108 ++ drivers/crypto/cmh/include/cmh_config.h | 105 ++ drivers/crypto/cmh/include/cmh_debugfs.h | 93 ++ drivers/crypto/cmh/include/cmh_dma.h | 249 +++ drivers/crypto/cmh/include/cmh_drbg_abi.h | 67 + drivers/crypto/cmh/include/cmh_eac_abi.h | 44 + drivers/crypto/cmh/include/cmh_hc_abi.h | 162 ++ drivers/crypto/cmh/include/cmh_hcq_abi.h | 222 +++ drivers/crypto/cmh/include/cmh_kic_abi.h | 77 + drivers/crypto/cmh/include/cmh_mqi.h | 35 + drivers/crypto/cmh/include/cmh_pke_abi.h | 272 ++++ drivers/crypto/cmh/include/cmh_qse_abi.h | 201 +++ drivers/crypto/cmh/include/cmh_registers.h | 161 ++ drivers/crypto/cmh/include/cmh_rh.h | 93 ++ drivers/crypto/cmh/include/cmh_sm3_abi.h | 79 + drivers/crypto/cmh/include/cmh_sm4_abi.h | 101 ++ drivers/crypto/cmh/include/cmh_sys_abi.h | 148 ++ drivers/crypto/cmh/include/cmh_sysfs.h | 14 + drivers/crypto/cmh/include/cmh_txn.h | 481 ++++++ drivers/crypto/cmh/include/cmh_vcq.h | 288 ++++ 38 files changed, 9361 insertions(+)
Documentation/ABI/testing/debugfs-driver-cmh new file +171 -0
@@ -0,0 +1,171 @@ +What: /sys/kernel/debug/cmh/mbx<N>/vcqs_submitted +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Total number of VCQ command entries submitted to + mailbox N since the driver was loaded. + +What: /sys/kernel/debug/cmh/mbx<N>/vcqs_completed +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Total number of VCQ command completions received + from mailbox N. + +What: /sys/kernel/debug/cmh/mbx<N>/vcqs_errors +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Total number of error completions received from + mailbox N. + +What: /sys/kernel/debug/cmh/mbx<N>/queue_full_count +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Number of times the transaction manager skipped + mailbox N because its in-flight queue was full. + +What: /sys/kernel/debug/cmh/mbx<N>/max_queue_depth +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) High-water mark of in-flight transactions on + mailbox N. + +What: /sys/kernel/debug/cmh/mbx<N>/inject_abort +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (WO) Write any value to inject an MBX_COMMAND_ABORT on + mailbox N. The abort triggers error-IRQ handling that + completes all in-flight transactions with -EIO and then + issues MBX_COMMAND_RESTART to resume the mailbox. + Only available when CONFIG_CRYPTO_DEV_CMH_DEBUG is enabled. + +What: /sys/kernel/debug/cmh/mbx<N>/force_drain +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (WO) Write any value to unconditionally FLUSH and drain + all pending transactions on mailbox N, completing each + with -ECANCELED, and reset all recovery bookkeeping + (including the wedged flag). The mailbox is re-enabled + for new work immediately; no hardware health verification + is performed. Use as a last-resort recovery when the eSW + is unresponsive and normal ABORT/RESTART escalation has + not recovered the mailbox. + Only available when CONFIG_CRYPTO_DEV_CMH_DEBUG is enabled. + +What: /sys/kernel/debug/cmh/tm/cmq_posts +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Total number of cmh_tm_post_command() calls (one + per crypto request submitted to the transaction manager). + +What: /sys/kernel/debug/cmh/tm/cmq_depth_max +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) High-water mark of the command queue length. + +What: /sys/kernel/debug/cmh/tm/cmq_eagain_count +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Number of times the command queue was full and + returned -EAGAIN to the caller. + +What: /sys/kernel/debug/cmh/tm/backoff_count +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Number of times the transaction manager backed off + because all mailbox queues were full. + +What: /sys/kernel/debug/cmh/tm/async_timeout_count +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RO) Number of async crypto requests that timed out + waiting for hardware completion. + +What: /sys/kernel/debug/cmh/config/async_timeout_ms +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RW) Async request timeout in milliseconds. On timeout + the driver issues MBX_COMMAND_ABORT; if the eSW is + unresponsive, the watchdog escalates through RESTART, + FLUSH, and force-drain to bound D-state duration. + +What: /sys/kernel/debug/cmh/config/vcq_timeout_ms +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RW) VCQ command timeout in milliseconds. + +What: /sys/kernel/debug/cmh/config/slow_op_timeout_ms +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RW) Slow-operation timeout in milliseconds. Used for + operations known to take longer (e.g. RSA key generation, + PQC key generation). + +What: /sys/kernel/debug/cmh/config/drain_timeout_ms +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RW) Drain timeout in milliseconds. Maximum time to wait + for all in-flight transactions to complete during driver + removal or suspend. + +What: /sys/kernel/debug/cmh/config/watchdog_ms +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RW) Result-handler watchdog interval in milliseconds. + Detects missed IRQs, stuck mailboxes, and abort-stall + conditions. Clamped to a 10 ms minimum. + +What: /sys/kernel/debug/cmh/config/drbg_timeout_ms +Date: June 2026 +KernelVersion: 7.1 +Contact: [email protected] +Description: + (RW) DRBG self-seed timeout in milliseconds. + +What: /sys/kernel/debug/cmh/config/cmq_max_depth +Date: August 2026 +KernelVersion: 7.2 +Contact: [email protected] +Description: + (RW) Maximum number of pending commands in the central + Command Message Queue. + +What: /sys/kernel/debug/cmh/config/backlog_max_depth +Date: August 2026 +KernelVersion: 7.2 +Contact: [email protected] +Description: + (RW) Maximum depth of the backlog queue for + CRYPTO_TFM_REQ_MAY_BACKLOG requests (0 disables backlog).
Documentation/ABI/testing/sysfs-driver-cmh new file +67 -0
67 lines not on this page · raw message
Documentation/crypto/device_drivers/cmh.rst new file +340 -0
340 lines not on this page · raw message
Documentation/crypto/device_drivers/index.rst +1 -0
@@ -6,4 +6,5 @@ Hardware Device Driver Specific Documentation .. toctree:: :maxdepth: 1 + cmh octeontx2
drivers/crypto/Kconfig +1 -0
@@ -831,4 +831,5 @@ source "drivers/crypto/starfive/Kconfig" source "drivers/crypto/inside-secure/eip93/Kconfig" source "drivers/crypto/ti/Kconfig" +source "drivers/crypto/cmh/Kconfig" endif # CRYPTO_HW
drivers/crypto/Makefile +1 -0
@@ -46,3 +46,4 @@ obj-y += intel/ obj-y += starfive/ obj-y += cavium/ obj-y += ti/ +obj-$(CONFIG_CRYPTO_DEV_CMH) += cmh/
drivers/crypto/cmh/Kconfig new file +62 -0
62 lines not on this page · raw message
drivers/crypto/cmh/Makefile new file +25 -0
@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for the Rambus CryptoManager Hub (CMH) hardware crypto accelerator driver. +# + +obj-$(CONFIG_CRYPTO_DEV_CMH) += cmh.o + +cmh-y := \ + cmh_main.o \ + cmh_config.o \ + cmh_mqi.o \ + cmh_txn.o \ + cmh_rh.o \ + cmh_dma.o \ + cmh_sysfs.o + +ccflags-y += -I$(src)/include + +# Suppress -Woverride-init for the [0 ... N] = -1 range-initializer pattern +# (standard kernel idiom for sparse lookup tables with a default value). +CFLAGS_cmh_config.o += -Wno-override-init + +# Debug instrumentation: per-mailbox debugfs counters. +# cmh_debugfs.o is linked into the composite cmh.o (same tristate). +cmh-$(CONFIG_CRYPTO_DEV_CMH_DEBUG) += cmh_debugfs.o
drivers/crypto/cmh/cmh_config.c new file +619 -0
619 lines not on this page · raw message
drivers/crypto/cmh/cmh_debugfs.c new file +307 -0
307 lines not on this page · raw message
drivers/crypto/cmh/cmh_dma.c new file +482 -0
482 lines not on this page · raw message
drivers/crypto/cmh/cmh_main.c new file +369 -0
369 lines not on this page · raw message
drivers/crypto/cmh/cmh_mqi.c new file +344 -0
344 lines not on this page · raw message
drivers/crypto/cmh/cmh_rh.c new file +1224 -0
200 lines not on this page · raw message
drivers/crypto/cmh/cmh_sysfs.c new file +108 -0
108 lines not on this page · raw message
drivers/crypto/cmh/cmh_txn.c new file +2118 -0
2,118 lines not on this page · raw message
drivers/crypto/cmh/include/cmh.h new file +24 -0
Lines of this file are not shown. View the raw message.
drivers/crypto/cmh/include/cmh_aes_abi.h new file +98 -0
98 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_ccp_abi.h new file +108 -0
108 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_config.h new file +105 -0
105 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_debugfs.h new file +93 -0
93 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_dma.h new file +249 -0
249 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_drbg_abi.h new file +67 -0
67 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_eac_abi.h new file +44 -0
44 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_hc_abi.h new file +162 -0
162 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_hcq_abi.h new file +222 -0
222 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_kic_abi.h new file +77 -0
77 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_mqi.h new file +35 -0
Lines of this file are not shown. View the raw message.
drivers/crypto/cmh/include/cmh_pke_abi.h new file +272 -0
272 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_qse_abi.h new file +201 -0
201 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_registers.h new file +161 -0
161 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_rh.h new file +93 -0
93 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_sm3_abi.h new file +79 -0
79 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_sm4_abi.h new file +101 -0
101 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_sys_abi.h new file +148 -0
148 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_sysfs.h new file +14 -0
Lines of this file are not shown. View the raw message.
drivers/crypto/cmh/include/cmh_txn.h new file +481 -0
481 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_vcq.h new file +288 -0
288 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.43.7
Part of the diff is not shown here. View the raw message.
[email protected]
AI reviewRe: [PATCH v7 2/5] crypto: cmh - add core platform driver
Thank you for your contribution! Sashiko AI review found 13 potential issue(s) to consider: - [Critical] crypto: cmh - uncancelled and unaborted sync timeouts in cmh_txn.c - [Critical] crypto: cmh…
Thank you for your contribution! Sashiko AI review found 13 potential issue(s) to consider:
- [Critical] crypto: cmh - uncancelled and unaborted sync timeouts in cmh_txn.c
- [Critical] crypto: cmh - duplicate MBX_ERROR_IRQ processing in cmh_rh_thread()
- [Critical] crypto: cmh - MBX_COMMAND_FLUSH clobbered by RESTART in cmh_rh.c
- [Critical] crypto: cmh - race between submit_vcq() and cmh_rh_drain_mbx()
- [High] crypto: cmh - premature backlog promotion in cmh_tm_thread()
- [High] crypto: cmh - cmh_rh_hardirq() clears R_MBX_INTERRUPT across all mailboxes
- [High] crypto: cmh - cmh_remove() frees DMA buffers before cmh_mqi_cleanup() flush
- [High] crypto: cmh - cmh_tm_quiesce() cancels in-flight txns without flushing HW
- [High] crypto: cmh: missing -EINPROGRESS notification when draining backlog
- [Medium] crypto: cmh - of_irq_get() 0 return treated as valid IRQ in cmh_config_init()
- [Medium] crypto: cmh - unlocked rh.mbx check in cmh_rh_abort_mbx() races teardown
- [Medium] crypto: cmh - dangling dbgfs.config pointer after cmh_debugfs_cleanup()
- [Low] crypto: cmh - missing stub for cmh_debugfs_config_u32() in cmh_debugfs.h --
commit fafcc004ae237081d15795f4adc63a0fe4baf892
Author: Alex Ousherovitch <[email protected]>
crypto: cmh - add core platform driver
This commit adds the Rambus CryptoManager Hub (CMH) hardware crypto accelerator core platform driver, including mailbox queue initialization, transaction management, threaded-IRQ response handling, DMA helpers, sysfs, and debugfs support.
Quoted 5 lines
drivers/crypto/cmh/cmh_config.c new file +0
-0
[ ... ]
Quoted 2 lines
drivers/crypto/cmh/cmh_config.c new file +2
-0+int cmh_config_init (struct cmh_config *cfg,struct platform_device *pdev) +{
[ ... ]
drivers/crypto/cmh/cmh_config.c line 506
Quoted 8 lines
drivers/crypto/cmh/cmh_config.c new file +8
-0+/* Optional per-mailbox interrupt (absent => polling). */ + m->irq =of_irq_get (child,0 ); +if (m->irq == -EPROBE_DEFER ) { +of_node_put (child); +return -EPROBE_DEFER ; + } +if (m->irq <0 ) + m->irq =-1 ;
[Severity: Medium]
What happens if of_irq_get() returns 0 on an IRQ mapping failure?
Because of_irq_get() returns 0 (NO_IRQ) when irq_create_of_mapping() fails, checking only m->irq < 0 leaves m->irq == 0. Later, cmh_rh_resolve_irqs() checks cfg->mailboxes[i].irq >= 0, treating 0 as a valid Linux IRQ and passing it to request_threaded_irq(0, ...) in cmh_rh_init(), which fails probe with -EINVAL instead of falling back to polling or handling the mapping failure.
Should cmh_config_init() treat m->irq <= 0 as no interrupt (or an error), and cmh_rh_resolve_irqs() check irq > 0?
Quoted 5 lines
drivers/crypto/cmh/cmh_debugfs.c new file +0
-0
[ ... ]
Quoted 24 lines
drivers/crypto/cmh/cmh_debugfs.c new file +24
-0+void cmh_debugfs_init (struct cmh_config *cfg) +{ +u32 mbx_count = cfg->mbx_count; +u32 i; + + dbgfs.root =debugfs_create_dir ("cmh" ,NULL ); +if (IS_ERR_OR_NULL (dbgfs.root)) { +if (!IS_ERR (dbgfs.root)) +dev_warn (cmh_dev (),"debugfs: creation returned NULL -- counters disabled\n" ); +else +dev_warn (cmh_dev (),"debugfs: creation failed (rc=%ld) -- counters disabled\n" , +PTR_ERR (dbgfs.root)); + dbgfs.root =NULL ; +return ;/* debugfs is optional -- never fail module init */ + } + + dbgfs.mbx_count = mbx_count; + dbgfs.cfg = cfg; + dbgfs.mbx =kcalloc (mbx_count,sizeof (*dbgfs.mbx),GFP_KERNEL ); +if (!dbgfs.mbx) { +debugfs_remove_recursive (dbgfs.root); + dbgfs.root =NULL ; +return ; + }
[ ... ]
Quoted 14 lines
drivers/crypto/cmh/cmh_debugfs.c new file +14
-0+void cmh_debugfs_cleanup (void ) +{ +debugfs_remove_recursive (dbgfs.root); + dbgfs.root =NULL ; +kfree (dbgfs.mbx); + dbgfs.mbx =NULL ; +dev_dbg (cmh_dev (),"debugfs: cleaned up\n" ); +} + +/** +* cmh_debugfs_config_u32() - Publish a subsystem tunable in config/ +* @name: debugfs file name +* @val: pointer to the u32 tunable (must outlive the debugfs tree) +*
drivers/crypto/cmh/cmh_debugfs.c line 282
Quoted 10 lines
drivers/crypto/cmh/cmh_debugfs.c new file +10
-0+* Lets a subsystem (e.g. the hwrng) register its own runtime tunable in +* the shared config/ directory as it comes up, so this core debugfs file +* never references per-algorithm symbols -- each algorithm driver stays +* self-contained. A no-op if debugfs is unavailable. +*/ +void cmh_debugfs_config_u32 (const char *name,u32 *val) +{ +if (dbgfs.config) +debugfs_create_u32 (name,0644 , dbgfs.config, val); +}
[Severity: Medium]
Does cmh_debugfs_cleanup() leave a dangling dentry pointer in dbgfs.config and a dangling config pointer in dbgfs.cfg?
debugfs_remove_recursive(dbgfs.root) frees the child dbgfs.config dentry along with dbgfs.root, but cmh_debugfs_cleanup() only clears dbgfs.root and dbgfs.mbx, leaving dbgfs.config, dbgfs.cfg, and dbgfs.mbx_count populated (and the kcalloc failure path in cmh_debugfs_init() likewise leaves dbgfs.cfg and dbgfs.mbx_count set).
If a subsequent probe fails in cmh_debugfs_init() before create_config_dir() runs, dbgfs.config still points to the freed dentry from the prior instance, so any call to cmh_debugfs_config_u32() will pass that freed dentry to debugfs_create_u32().
Quoted 5 lines
drivers/crypto/cmh/cmh_main.c new file +0
-0
[ ... ]
Quoted 6 lines
drivers/crypto/cmh/cmh_main.c new file +6
-0+static void cmh_remove (struct platform_device *pdev) +{ +struct cmh_device *dev =platform_get_drvdata (pdev); +struct cmh_config *cfg; + +if (!dev)
drivers/crypto/cmh/cmh_main.c line 263
Quoted 10 lines
drivers/crypto/cmh/cmh_main.c new file +10
-0+return ; + + cfg = &dev->config; + +cmh_rh_cleanup (cfg); +cmh_tm_cleanup (); +cmh_mqi_cleanup (cfg); +cmh_debugfs_cleanup (); +cmh_dma_cleanup (); +}
[Severity: High]
In cmh_remove(), can calling cmh_tm_cleanup() before cmh_mqi_cleanup(cfg) unmap and free DMA buffers while the hardware mailboxes are still running?
cmh_rh_cleanup(cfg) masks interrupts and frees the IRQ handlers, but leaves each mailbox in MBX_COMMAND_RUN state. Next, cmh_tm_cleanup() force-completes all transactions in tm.txqs (causing completion callbacks to unmap and free per-request DMA buffers) and calls cmh_dma_reclaim_parked() (which unmaps and frees all parked orphan DMA buffers).
Only after those DMA buffers have been freed does cmh_mqi_cleanup(cfg) -> cmh_mbx_teardown_one() write MBX_COMMAND_FLUSH to R_MBX_COMMAND and poll for the eSW to finish flushing the mailboxes. Should the mailboxes be flushed before cmh_tm_cleanup() unmaps and frees the in-flight and parked DMA buffers?
Quoted 5 lines
drivers/crypto/cmh/cmh_rh.c new file +0
-0
[ ... ]
drivers/crypto/cmh/cmh_rh.c line 261
Quoted 13 lines
drivers/crypto/cmh/cmh_rh.c new file +13
-0+void cmh_rh_abort_mbx (u32 mbx_idx) +{ +void __iomem *base; + +if (!rh.cfg || !rh.mbx || mbx_idx >= rh.cfg->mbx_count) +return ; + + base = rh.cfg->mailboxes[mbx_idx].reg_base; + +spin_lock_bh (&rh_process_lock); +cmh_reg_write32 (MBX_COMMAND_ABORT , base,R_MBX_COMMAND ); +spin_unlock_bh (&rh_process_lock); +}
[Severity: Medium]
Unlike cmh_rh_force_drain_mbx() right above it, cmh_rh_abort_mbx() checks rh.cfg and rh.mbx outside rh_process_lock.
Because cmh_remove() and the cmh_probe() error path tear down RH, TM, and MQI before calling cmh_debugfs_cleanup(), can a concurrent write to the debugfs inject_abort file pass the unlocked check just before cmh_rh_cleanup() sets rh.mbx = NULL, and then write MBX_COMMAND_ABORT to R_MBX_COMMAND after the mailbox has been torn down and unlocked?
Quoted 15 lines
drivers/crypto/cmh/cmh_rh.c new file +15
-0+static struct timer_list rh_watchdog; + +/* +* Hardirq handler -- runs with interrupts disabled. +* +* Read and W1C-clear R_MBX_INTERRUPT for each mailbox. +* If any MBX had a pending interrupt, return IRQ_WAKE_THREAD. +* Shared-IRQ safe: returns IRQ_NONE if we didn't handle anything. +*/ +static irqreturn_t cmh_rh_hardirq (int irq,void *data) +{ +struct cmh_config *cfg = data; +bool handled = false; +u32 i; +
drivers/crypto/cmh/cmh_rh.c line 287
Quoted 10 lines
drivers/crypto/cmh/cmh_rh.c new file +10
-0+for (i =0 ; i < cfg->mbx_count; i++) { +void __iomem *base = cfg->mailboxes[i].reg_base; +u32 bits; + + bits =cmh_reg_read32 (base,R_MBX_INTERRUPT ); +if (!bits) +continue ; + +/* W1C: write back the set bits to clear them */ +cmh_reg_write32 (bits, base,R_MBX_INTERRUPT );
[Severity: High]
When multiple mailboxes have distinct IRQ lines (rh.nirqs > 1), what happens when cmh_rh_hardirq() ignores the irq parameter and W1C-clears R_MBX_INTERRUPT on every mailbox?
If mailbox 0 and mailbox 1 both assert their separate interrupt lines at the same time, the hardirq handler for mailbox 0's IRQ line will clear R_MBX_INTERRUPT on both mailbox 0 and mailbox 1. When the hardirq handler for mailbox 1's IRQ line runs, it will read 0 from R_MBX_INTERRUPT on all mailboxes and return IRQ_NONE, which can trip the kernel's spurious IRQ detector ("nobody cared") and disable mailbox 1's interrupt line.
Should cmh_rh_hardirq() only inspect and W1C-clear mailboxes whose configured irq matches the irq argument?
[ ... ]
Quoted 2 lines
drivers/crypto/cmh/cmh_rh.c new file +2
-0+static void cmh_rh_process_mbx (u32 mbx_idx,u32 new_head,u32 irq_bits) +{
[ ... ]
drivers/crypto/cmh/cmh_rh.c line 428
Quoted 13 lines
drivers/crypto/cmh/cmh_rh.c new file +13
-0+ txn =cmh_tm_peek_transaction (mbx_idx); +if (txn && txn->first_vcq_id != txn->last_vcq_id) { +dev_warn_ratelimited (cmh_dev (), +"rh: mbx[%u] multi-VCQ error vcq=%u..%u -- FLUSH+drain\n" , + mbx_idx, txn->first_vcq_id, + txn->last_vcq_id); +cmh_rh_stat_inc_errors (mbx_idx); +cmh_reg_write32 (MBX_IRQ_MASK , base,R_MBX_INTERRUPT ); +cmh_reg_write32 (MBX_COMMAND_FLUSH , base,R_MBX_COMMAND ); +cmh_rh_poke_tail (base); +cmh_rh_drain_mbx (mbx_idx, error); +return ; + }
[Severity: Critical]
When a multi-VCQ transaction fails here (or when cmh_rh_watchdog_fn() escalates a timed-out RESTART to MBX_COMMAND_FLUSH), can the asynchronous MBX_COMMAND_FLUSH get immediately overwritten by MBX_COMMAND_RESTART?
In cmh_rh_process_mbx():
1. For a multi-VCQ error, it writes MBX_COMMAND_FLUSH to R_MBX_COMMAND, calls cmh_rh_drain_mbx() (which completes all queued transactions, frees their DMA buffers, and updates rh.mbx[mbx_idx].last_head to the hardware HEAD), and returns without setting rh.mbx[mbx_idx].restart_pending.
2. All three callers of cmh_rh_process_mbx() (both loops in cmh_rh_thread() and cmh_rh_watchdog_fn()) then execute:
cmh_rh_process_mbx(i, new_head, ...);
rh.mbx[i].last_head = new_head; overwriting the post-drain last_head with the stale pre-flush new_head.
3. When this runs from the first loop of cmh_rh_thread(), execution immediately enters the final hardware poll loop in cmh_rh_thread(). Because the eSW processes MBX_COMMAND_FLUSH asynchronously, R_MBX_STATUS still reports MBX_STATUS_ERROR and restart_pending is false. The final poll loop therefore sets poll_irq_bits = MBX_ERROR_IRQ and calls cmh_rh_process_mbx() a second time on the now-empty queue, which falls through to the single-VCQ path and writes MBX_COMMAND_RESTART to R_MBX_COMMAND.
As the comment just above notes, RESTART only advances HEAD by one slot instead of discarding the queue, so clobbering FLUSH with RESTART causes the eSW to execute the remaining slots of the multi-VCQ transaction whose DMA buffers were just freed by cmh_rh_drain_mbx(). The same issue occurs in cmh_rh_watchdog_fn() when restart_retries exceeds CMH_RH_RESTART_MAX_RETRIES, because it clears restart_pending immediately after writing MBX_COMMAND_FLUSH.
Quoted 10 lines
drivers/crypto/cmh/cmh_rh.c new file +10
-0+ +/* Now pop and fail the transaction that actually errored */ + txn =cmh_tm_pop_transaction (mbx_idx); +if (txn) { +dev_dbg (cmh_dev (),"rh: mbx[%u] error-complete vcq=%u..%u\n" , + mbx_idx, txn->first_vcq_id, + txn->last_vcq_id); +cmh_txn_finish (txn, error); +cmh_tm_txq_completion_notify (); + }else {
[ ... ]
Quoted 29 lines
drivers/crypto/cmh/cmh_rh.c new file +29
-0+static irqreturn_t cmh_rh_thread (int irq,void *data) +{ +struct cmh_config *cfg = data; +u32 i; +bool recheck; + + do { + recheck = false; + + for (i = 0; i < cfg->mbx_count; i++) { + u32 new_head, irq_bits; + + if (!READ_ONCE(rh.mbx[i].pending)) + continue; + + /* + * Clear @pending before consuming @irq_bits so a + * hardirq landing between the two does not have its + * freshly-set @pending cleared and its bits stranded. + * The xchg is a full barrier and captures any bits the + * hardirq ORs in; @pending then stays set and the outer + * loop rescans this mailbox. + */ + WRITE_ONCE(rh.mbx[i].pending, false); + irq_bits = (u32)atomic_xchg(&rh.mbx[i].irq_bits, 0); + + spin_lock_bh(&rh_process_lock); + new_head = cmh_reg_read32(cfg->mailboxes[i].reg_base, + R_MBX_QUEUE_HEAD);
drivers/crypto/cmh/cmh_rh.c line 588
Quoted 10 lines
drivers/crypto/cmh/cmh_rh.c new file +10
-0+ +if (new_head == rh.mbx[i].last_head && !irq_bits) { +spin_unlock_bh (&rh_process_lock); +continue ; + } + +cmh_rh_process_mbx (i, new_head, irq_bits); + rh.mbx[i].last_head = new_head; +spin_unlock_bh (&rh_process_lock); + }
[Severity: Critical]
Can a single hardware mailbox error be processed twice between cmh_rh_hardirq()/cmh_rh_thread() and the status-polling paths in cmh_rh_watchdog_fn() or cmh_rh_thread()'s final poll loop?
Consider this sequence when a VCQ fails in hardware:
1. The eSW sets MBX_ERROR_IRQ in R_MBX_INTERRUPT and sets R_MBX_STATUS to MBX_STATUS_ERROR.
2. cmh_rh_hardirq() W1C-clears R_MBX_INTERRUPT, ORs MBX_ERROR_IRQ into rh.mbx[i].irq_bits, and sets rh.mbx[i].pending = true.
3. Before cmh_rh_thread()'s first loop acquires rh_process_lock (or while cmh_rh_thread() is in its final poll loop when cmh_rh_hardirq() fires on another CPU), cmh_rh_watchdog_fn() or the final poll loop acquires rh_process_lock, sees MBX_STATUS_CODE(status) == MBX_STATUS_ERROR with restart_pending == false, synthesizes MBX_ERROR_IRQ, and calls cmh_rh_process_mbx(i, new_head, MBX_ERROR_IRQ).
4. cmh_rh_process_mbx() pops the failed transaction, completes it with -EIO, writes MBX_COMMAND_RESTART, and sets rh.mbx[i].restart_pending = true, without clearing rh.mbx[i].irq_bits or rh.mbx[i].pending.
5. When cmh_rh_thread()'s primary loop runs next, it consumes the latched MBX_ERROR_IRQ from rh.mbx[i].irq_bits, does not check rh.mbx[i].wedged or rh.mbx[i].restart_pending, and calls cmh_rh_process_mbx(i, new_head, MBX_ERROR_IRQ) a second time for the same error.
Because cmh_rh_process_mbx() unconditionally pops the head of txq without checking whether txn->first_vcq_id <= new_head, this second call pops and fails the next innocent in-flight transaction with -EIO (unmapping and freeing its DMA buffers) just as the eSW processes MBX_COMMAND_RESTART and starts executing that transaction's VCQ.
Quoted 5 lines
drivers/crypto/cmh/cmh_txn.c new file +0
-0
[ ... ]
Quoted 2 lines
drivers/crypto/cmh/cmh_txn.c new file +2
-0+static int submit_vcq (struct command_msg *msg,u32 mbx_idx) +{
[ ... ]
Quoted 21 lines
drivers/crypto/cmh/cmh_txn.c new file +21
-0+if (msg->timeout_jiffies) +refcount_set (&txn->refs,2 ); +else +refcount_set (&txn->refs,1 ); + +/* Enqueue transaction under spinlock */ +spin_lock_irqsave (&txq->lock, flags); +list_add_tail (&txn->list, &txq->head); + txq->depth++; +spin_unlock_irqrestore (&txq->lock, flags); + +/* +* Arm the per-request timeout BEFORE the doorbell (async only): once +* the doorbell rings, HW can complete and the TXQ can pop and free the +* txn concurrently, so arming afterwards races that free and can leave +* timer_delete() in cmh_txn_finish() unable to drop the timer ref. +*/ +if (msg->timeout_jiffies) +mod_timer (&txn->timeout_timer, + jiffies + msg->timeout_jiffies); +
drivers/crypto/cmh/cmh_txn.c line 861
Quoted 10 lines
drivers/crypto/cmh/cmh_txn.c new file +10
-0+/* +* Ring doorbell: advance tail by number of VCQs submitted. +* Serialise against cmh_tm_poke_tail() so a concurrent re-poke +* cannot roll TAIL back over this advance. +*/ +spin_lock_irqsave (&cmh_mbx_tail_lock, flags); +cmh_reg_write32 (tail + num_vcqs, mbx->reg_base,R_MBX_QUEUE_TAIL ); +spin_unlock_irqrestore (&cmh_mbx_tail_lock, flags); + +mutex_unlock (&txq->dispatch_lock);
[Severity: Critical]
Can submit_vcq() race with cmh_rh_drain_mbx() after txq->lock is dropped here?
submit_vcq() holds only txq->dispatch_lock, while cmh_rh_drain_mbx() (called on multi-VCQ error, watchdog FLUSH/abort-stall escalation, or debugfs force_drain) holds only rh_process_lock. Once txn is added to txq->head and txq->lock is released:
1. cmh_rh_drain_mbx() can pop txn from txq->head and call cmh_txn_finish() before mod_timer() and the R_MBX_QUEUE_TAIL doorbell write execute.
2. In cmh_txn_finish(), timer_delete(&txn->timeout_timer) returns false because mod_timer() has not run yet, so the timer reference is not dropped; for transactions with msg->timeout_jiffies == 0 (refs == 1), cmh_txn_finish() frees txn immediately. In both cases txn->complete() fires and unmaps/frees the caller's DMA buffers.
3. submit_vcq() then calls mod_timer(&txn->timeout_timer, ...) on the already-completed (or freed) txn, leaving an active timer on a dequeued transaction, and writes tail + num_vcqs to R_MBX_QUEUE_TAIL so the hardware executes a VCQ whose DMA buffers were already freed.
Quoted 67 lines
drivers/crypto/cmh/cmh_txn.c new file +67
-0+ +cmh_stat_record_vcq_submit (mbx_idx, num_vcqs, txq->depth); + +dev_dbg (cmh_dev (),"tm: submitted %u vcq(s) id=%u..%u to mbx[%u] tail_now=%u\n" , + num_vcqs, tail, tail + num_vcqs -1 , mbx_idx, + tail + num_vcqs); + +return 0 ; +} + +/* TM Thread */ + +static int cmh_tm_thread (void *data) +{ +struct command_msg *msg, *bl_promoted; +unsigned long flags; +int mbx_idx, ret; + +while (!kthread_should_stop ()) { +/* Wait for work or stop signal */ +wait_event_interruptible (tm.cmq_waitq, + !list_empty (&tm.cmq) ||kthread_should_stop ()); + +if (kthread_should_stop ()) +break ; + +/* Dequeue one command message */ +spin_lock_irqsave (&tm.cmq_lock, flags); +if (list_empty (&tm.cmq)) { +spin_unlock_irqrestore (&tm.cmq_lock, flags); +continue ; + } + msg = list_first_entry(&tm.cmq, struct command_msg, list); +list_del_init (&msg->list); + cmq_depth--; + +/* +* Promote one backlogged request into the CMQ now that +* there is room. The -EINPROGRESS notification is deferred +* until after cmq_lock is dropped (below). +*/ + bl_promoted =NULL ; +if (!list_empty (&tm.backlog)) { + bl_promoted =list_first_entry (&tm.backlog, + struct command_msg, list); +list_move_tail (&bl_promoted->list, &tm.cmq); + tm.backlog_depth--; + cmq_depth++; +cmh_stat_record_cmq_post (cmq_depth); + } + +spin_unlock_irqrestore (&tm.cmq_lock, flags); + +/* +* Signal -EINPROGRESS for the promoted backlog request with +* cmq_lock dropped: a consumer that resubmits from this +* callback would otherwise re-enter cmq_lock and self-deadlock. +* The promoted msg stays on the CMQ (only this thread dequeues +* it) until a later iteration, so the -EINPROGRESS still +* precedes its final completion. +*/ +if (bl_promoted && bl_promoted->complete) + bl_promoted->complete (bl_promoted->completion_data, + -EINPROGRESS ); + +/* Select a mailbox: pinned or round-robin */ + mbx_idx =resolve_mbx (msg);
[ ... ]
Quoted 7 lines
drivers/crypto/cmh/cmh_txn.c new file +7
-0+if (mbx_idx <0 ) { +/* +* Queue full -- re-enqueue at front and wait. +* +* Sleep on cmq_waitq with a short timeout. The RH +* calls cmh_tm_txq_completion_notify() after each +* completed transaction, which bumps the generation
drivers/crypto/cmh/cmh_txn.c line 978
Quoted 10 lines
drivers/crypto/cmh/cmh_txn.c new file +10
-0+* counter and wakes us immediately. The timeout is +* a safety net for missed wakeups. +*/ +int gen =atomic_read (&txq_completion_gen); +unsigned long tmo; + +spin_lock_irqsave (&tm.cmq_lock, flags); +list_add (&msg->list, &tm.cmq); + cmq_depth++; +spin_unlock_irqrestore (&tm.cmq_lock, flags);
[Severity: High]
What happens to cmq_depth and tm.backlog when resolve_mbx(msg) returns -EAGAIN?
Before calling resolve_mbx(msg), cmh_tm_thread() pops msg (cmq_depth--), unconditionally promotes a request from tm.backlog into tm.cmq (cmq_depth++) without checking cmq_depth < cmq_max_depth, and invokes its completion callback with -EINPROGRESS. When all eligible mailboxes are full and resolve_mbx(msg) returns -EAGAIN, msg is pushed back onto the front of tm.cmq and cmq_depth is incremented again.
On each subsequent 500 us backoff wakeup (or completion wakeup from another mailbox while msg is pinned), cmh_tm_thread() pops msg again, promotes another entry from tm.backlog into tm.cmq, fires -EINPROGRESS (allowing throttled callers such as dm-crypt to submit more requests into tm.backlog), and re-adds msg to tm.cmq, growing tm.cmq without bound past cmq_max_depth.
Also, if queued entries in tm.cmq are cancelled via cmh_tm_try_cancel_command() (or cmq_max_depth is set to 0 via debugfs) while tm.backlog is non-empty, cmh_tm_thread() only waits on !list_empty(&tm.cmq), leaving tm.backlog stranded. Should backlog promotion happen only after msg is actually dispatched (or when cmq_depth < cmq_max_depth)?
[ ... ]
Quoted 2 lines
drivers/crypto/cmh/cmh_txn.c new file +2
-0+static void cmh_tm_stop_and_drain_cmq (void ) +{
[ ... ]
Quoted 6 lines
drivers/crypto/cmh/cmh_txn.c new file +6
-0+for (;;) { +spin_lock_irqsave (&tm.cmq_lock, flags); + msg = list_first_entry_or_null(&tm.backlog, struct command_msg, + list); +if (msg) { +list_del_init (&msg->list);
drivers/crypto/cmh/cmh_txn.c line 1129
Quoted 10 lines
drivers/crypto/cmh/cmh_txn.c new file +10
-0+ tm.backlog_depth--; + } +spin_unlock_irqrestore (&tm.cmq_lock, flags); +if (!msg) +break ; +if (msg->complete) + msg->complete (msg->completion_data, -ECANCELED ); +command_msg_put (msg); + } +}
[Severity: High]
When draining tm.backlog here, does msg->complete() need to be called with -EINPROGRESS before -ECANCELED?
Requests on tm.backlog returned -EBUSY from cmh_tm_post_command() and have not yet received their initial -EINPROGRESS notification from cmh_tm_thread(). Crypto API callers that use CRYPTO_TFM_REQ_MAY_BACKLOG (such as dm-crypt and dm-integrity) block in wait_for_completion() waiting specifically for the callback with err == -EINPROGRESS before continuing; completing a backlogged request only once with -ECANCELED leaves those waiters hung in TASK_UNINTERRUPTIBLE.
[ ... ]
Quoted 2 lines
drivers/crypto/cmh/cmh_txn.c new file +2
-0+void cmh_tm_quiesce (void ) +{
[ ... ]
Quoted 46 lines
drivers/crypto/cmh/cmh_txn.c new file +46
-0+if (!drained) { +dev_warn (cmh_dev (), +"tm: quiesce drain timeout (%u ms), cancelling remaining transactions\n" , + drain_timeout_ms); +/* +* The drain timed out, so the RH is no longer making +* progress. Quiesce it first -- cmh_rh_suspend() stops the +* watchdog, masks the MBX interrupts and synchronize_irq()s +* out any in-flight handler -- so the force-cancel below owns +* every remaining transaction exclusively. Without this a +* live RH could concurrently finish/txn_put a transaction we +* are cancelling here (suspend runs cmh_tm_quiesce() before +* cmh_rh_suspend()). cmh_rh_suspend() is idempotent, so the +* caller's later call is a no-op. +*/ +cmh_rh_suspend (tm.cfg); + +/* +* Force-cancelling with -ECANCELED/-ETIMEDOUT parks any +* abnormally-completed DRBG orphan buffer (see +* cmh_dma_orphan_free()). Parked buffers are NOT reclaimed on +* suspend -- the eSW may still be powered/writing -- so they +* persist across resume and are freed only at module unload. +*/ +for (i =0 ; i < tm.cfg->mbx_count; i++) { +LIST_HEAD (drain); +int old; + +spin_lock_irqsave (&tm.txqs[i].lock, flags); +list_splice_init (&tm.txqs[i].head, &drain); + tm.txqs[i].depth =0 ; +spin_unlock_irqrestore (&tm.txqs[i].lock, flags); + +list_for_each_entry_safe (txn, tmp_txn, &drain, list) { +list_del (&txn->list); + +if (timer_delete_sync (&txn->timeout_timer)) +txn_put (txn); + + old =atomic_cmpxchg (&txn->state, +TXN_INFLIGHT , +TXN_COMPLETE ); +if (txn->complete) { +if (old ==TXN_INFLIGHT ) + txn->complete (txn->completion_data, + -ECANCELED );
drivers/crypto/cmh/cmh_txn.c line 1307
Quoted 10 lines
drivers/crypto/cmh/cmh_txn.c new file +10
-0+else if (old ==TXN_TIMED_OUT ) + txn->complete (txn->completion_data, + -ETIMEDOUT ); + } + +txn_put (txn); + } + } + } +}
[Severity: High]
When drain_timeout_ms expires during system suspend in cmh_tm_quiesce(), is it safe to force-complete in-flight transactions without issuing MBX_COMMAND_FLUSH to the hardware mailboxes?
For async crypto requests, invoking txn->complete() causes the caller to immediately unmap and free its per-request DMA buffers, while the CMH island remains powered across suspend-to-idle and the eSW continues executing the un-flushed VCQs against those freed buffers. In addition, on resume, cmh_rh_resume() re-syncs last_head and re-enables interrupts without checking R_MBX_STATUS or flushing pre-suspend VCQs, so a pre-suspend VCQ that errors while interrupts were masked (or after resume) will cause cmh_rh_process_mbx() to pop and fail a newly submitted post-resume transaction.
[ ... ]
Quoted 4 lines
drivers/crypto/cmh/cmh_txn.c new file +4
-0+int cmh_tm_submit_sync_tmo (struct vcq_cmd *vcq_cmds,u32 vcq_count, +u32 num_vcqs,s32 target_mbx, +unsigned long timeout_hz) +{
[ ... ]
Quoted 72 lines
drivers/crypto/cmh/cmh_txn.c new file +72
-0+ left =wait_for_completion_timeout (&sync->done, timeout_hz); +if (!left) { +dev_err (cmh_dev (), +"tm: submit_sync timeout (%lums) cmd=0x%08x\n" , + timeout_hz *1000 /HZ , msg->command_id); +if (cmh_tm_try_cancel_command (msg)) { +/* +* Msg was still queued -- TM never saw it. +* Drop the callback ref (no txn will fire it) +* and free msg directly (sole owner). +*/ +cmh_sync_ctx_put (sync);/* no txn -> drop cb ref */ +cmh_sync_ctx_put (sync);/* drop waiter ref */ +command_msg_put (msg);/* matches refcount_set(2) */ +command_msg_put (msg); + }else { +/* +* TM has dequeued msg and the VCQ is in-flight. +* Issue MBX_COMMAND_ABORT to force-stop the VCQ; +* the RH will fire MBX_ERROR_IRQ, complete the +* transaction with -EIO, and issue RESTART. +* +* cmh_rh_abort_mbx() serialises the write under +* rh_process_lock, preventing clobber of a +* concurrent RESTART/FLUSH from the watchdog. +*/ +s32 abrt_mbx =READ_ONCE (msg->actual_mbx); + +/* +* Only ABORT if our VCQ is the one currently at the +* mailbox HEAD. ABORT stops whatever executes at +* HEAD, so aborting while a different (older, +* possibly long-running) transaction is at HEAD would +* kill an unrelated request. If ours is queued behind +* it, the HW has not started ours yet -- leave it to +* the completion wait / watchdog rather than abort +* someone else. +*/ +if (abrt_mbx >=0 && + (u32 )abrt_mbx < tm.cfg->mbx_count) { +void __iomem *abase = + tm.cfg->mailboxes[abrt_mbx].reg_base; +u32 head =cmh_reg_read32 (abase, +R_MBX_QUEUE_HEAD ); + +if (READ_ONCE (msg->first_vcq_id) == head) { +dev_warn (cmh_dev (), +"tm: aborting mbx[%d] cmd=0x%08x\n" , + abrt_mbx, msg->command_id); +cmh_rh_abort_mbx ((u32 )abrt_mbx); + } + } + +/* +* Wait for the RH completion (ABORT triggers +* MBX_ERROR_IRQ within microseconds). Fixed +* 5 s ceiling -- not configurable because if +* ABORT doesn't complete in this window the +* HW is wedged and more waiting won't help. +*/ + left =wait_for_completion_timeout (&sync->done, +5 *HZ ); +if (!left) { +/* +* ABORT did not complete within 5 s. Unlike +* cmh_tm_submit_sync_noabort, this path sets no +* orphan_cb, so the caller frees its DMA buffers +* on the -ETIMEDOUT below. This HW exposes no +* DMA-stop primitive, so where an ABORT cannot be +* issued or observed to complete the driver has +* no positive confirmation the engine has +* finished with those buffers.
drivers/crypto/cmh/cmh_txn.c line 1642
Quoted 10 lines
drivers/crypto/cmh/cmh_txn.c new file +10
-0+*/ +dev_err (cmh_dev (), +"tm: abort timeout (5s) cmd=0x%08x - HW wedged\n" , + msg->command_id); + } +cmh_sync_ctx_put (sync);/* drop waiter ref */ +command_msg_put (msg);/* drop waiter ref on msg */ + } +return -ETIMEDOUT ; + }
[Severity: Critical]
Can a synchronous submission in cmh_tm_submit_sync_tmo() or cmh_tm_submit_sync_noabort() fail to cancel or abort on timeout while the VCQ is still queued, leading to hardware DMA against freed caller buffers or a permanently wedged mailbox?
There are three related cases here:
1. In cmh_tm_thread(), msg is removed from tm.cmq with list_del_init(&msg->list) and tm.cmq_lock is dropped before resolve_mbx(msg) runs; if resolve_mbx(msg) returns -EAGAIN, msg is re-added to tm.cmq. If wait_for_completion_timeout() expires during that unlocked window, cmh_tm_try_cancel_command(msg) sees list_empty(&msg->list) and returns false, after which cmh_tm_thread() puts msg back on tm.cmq. cmh_tm_submit_sync_noabort() then returns -EINPROGRESS while msg is still in tm.cmq, and cmh_tm_submit_sync_tmo() sees abrt_mbx == -1, skips cmh_rh_abort_mbx(), and returns -ETIMEDOUT after 5s while msg remains queued.
2. When msg has been submitted to a mailbox (abrt_mbx >= 0), cmh_tm_submit_sync_tmo() only calls cmh_rh_abort_mbx() if READ_ONCE(msg->first_vcq_id) == head. If msg is queued behind a slow operation (for example, a 300s slow_op_timeout_ms operation at head while msg has a 2s vcq_timeout_ms, so head < msg->first_vcq_id), or if msg is a multi-VCQ submission stalled on a subsequent VCQ slot (head > msg->first_vcq_id), cmh_rh_abort_mbx() is skipped. After 5s, cmh_tm_submit_sync_tmo() returns -ETIMEDOUT and the caller unmaps and frees its DMA buffers (including DMA_TO_DEVICE input buffers that are never parked), after which the hardware executes msg's VCQ against freed memory.
3. Both cmh_tm_submit_sync_tmo() and cmh_tm_submit_sync_noabort() leave msg->timeout_jiffies == 0, so submit_vcq() never arms txn->timeout_timer and txn->state stays TXN_INFLIGHT forever. Because the abort-stall detector in cmh_rh_watchdog_fn() only triggers when atomic_read(&head_txn->state) == TXN_TIMED_OUT, the watchdog will never recover a synchronous transaction (such as an abandoned noabort VCQ) that stalls at the head of a mailbox.
Quoted 5 lines
drivers/crypto/cmh/include/cmh_debugfs.h new file +0
-0
[ ... ]
Quoted 9 lines
drivers/crypto/cmh/include/cmh_debugfs.h new file +9
-0+#ifdef CONFIG_CRYPTO_DEV_CMH_DEBUG + +void cmh_debugfs_init (struct cmh_config *cfg); +void cmh_debugfs_cleanup (void ); + +/* Publish a subsystem's u32 tunable in the shared config/ dir. */ +void cmh_debugfs_config_u32 (const char *name,u32 *val); + +struct cmh_mbx_stats *cmh_debugfs_mbx_stats (u32 mbx_idx);
drivers/crypto/cmh/include/cmh_debugfs.h line 92
Quoted 10 lines
drivers/crypto/cmh/include/cmh_debugfs.h new file +10
-0+struct cmh_tm_stats *cmh_debugfs_tm_stats (void ); + +#else /* !CONFIG_CRYPTO_DEV_CMH_DEBUG */ + +static inline void cmh_debugfs_init (struct cmh_config *c) { } +static inline void cmh_debugfs_cleanup (void ) {} +static inline struct cmh_mbx_stats *cmh_debugfs_mbx_stats (u32 i) {return NULL ; } +static inline struct cmh_tm_stats *cmh_debugfs_tm_stats (void ) {return NULL ; } + +#endif /* CONFIG_CRYPTO_DEV_CMH_DEBUG */
[Severity: Low]
Is a static inline stub for cmh_debugfs_config_u32() missing in the !CONFIG_CRYPTO_DEV_CMH_DEBUG branch so callers compile when debugfs support is disabled?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v7 3/5 crypto: cmh - add key/datastore infrastructure
In reply to Alex OusherovitchAdd the CMH key and datastore (system object) layer shared by the algorithm drivers. cmh_sys.c manages CMH system objects (allocate and free datastore slots); cmh_key.c stages a raw key into a per-transform context and DMA-maps it for the lifetime of the transform.
These are used by the symmetric, hash/MAC and asymmetric algorithm drivers that follow, and by the /dev/cmh_mgmt ioctl device. No crypto API algorithm is registered by this patch.
--- drivers/crypto/cmh/Makefile | 4 +- drivers/crypto/cmh/cmh_key.c | 130 ++++++++++++ drivers/crypto/cmh/cmh_sys.c | 376 +++++++++++++++++++++++++++++++++++ drivers/crypto/cmh/include/cmh_key.h | 73 +++++++ drivers/crypto/cmh/include/cmh_sys.h | 111 +++++++++++ 5 files changed, 693 insertions(+), 1 deletion(-)
drivers/crypto/cmh/Makefile +3 -1
@@ -12,7 +12,9 @@ cmh-y := \ cmh_txn.o \ cmh_rh.o \ cmh_dma.o \ - cmh_sysfs.o + cmh_sysfs.o \ + cmh_sys.o \ + cmh_key.o ccflags-y += -I$(src)/include
drivers/crypto/cmh/cmh_key.c new file +130 -0
130 lines not on this page · raw message
drivers/crypto/cmh/cmh_sys.c new file +376 -0
376 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_key.h new file +73 -0
73 lines not on this page · raw message
drivers/crypto/cmh/include/cmh_sys.h new file +111 -0
111 lines not on this page · raw message
Signature · 1 line
2.43.7
PATCH v7 4/5 crypto: cmh - add AES skcipher/aead/cmac
In reply to Alex OusherovitchRegister AES algorithms using the CMH AES core (core ID 0x03):
- skcipher: AES-ECB, AES-CBC, AES-CTR, AES-XTS, AES-CFB
- aead: AES-GCM, AES-CCM
- ahash: AES-CMAC
Supports 128, 192, and 256-bit keys. AEAD algorithms handle associated data, payload, and authentication tag with correct encrypt/decrypt separation.
AES-CMAC hardware-accelerates the one-shot .digest() only; the incremental and virtual-address (CRYPTO_AHASH_REQ_VIRT) paths are forwarded to a generic cmac(aes) transform the driver allocates in .init_tfm(). This is the one algorithm here that setsCRYPTO_ALG_NO_FALLBACK: crypto/cmac.c implements only descsize (no export_core/import_core), so the core marks it NO_EXPORT_CORE and crypto_ahash_init_tfm() cannot install it as the automatic crypto_ahash_fb() fallback. ahash_prepare_alg() then forces NEED_FALLBACK unless NO_FALLBACK is set, so the flag is required to instantiate an async digest-only CMAC at all; the driver allocates the generic transform itself to serve the non-digest paths. This is the same narrow use of NO_FALLBACK as s390; the hashes and HMAC in a later series use the core's crypto_ahash_fb() and do not set it.
--- drivers/crypto/cmh/Makefile | 5 +- drivers/crypto/cmh/cmh_aes.c | 755 +++++++++++++++++++++++++ drivers/crypto/cmh/cmh_aes_aead.c | 1013 ++++++++++++++++++++++++++++++++++ drivers/crypto/cmh/cmh_aes_cmac.c | 476 ++++++++++++++++ drivers/crypto/cmh/cmh_main.c | 25 + drivers/crypto/cmh/include/cmh_aes.h | 24 + 6 files changed, 2297 insertions(+), 1 deletion(-)
drivers/crypto/cmh/Makefile +4 -1
@@ -14,7 +14,10 @@ cmh-y := \ cmh_dma.o \ cmh_sysfs.o \ cmh_sys.o \ - cmh_key.o + cmh_key.o \ + cmh_aes.o \ + cmh_aes_aead.o \ + cmh_aes_cmac.o ccflags-y += -I$(src)/include
drivers/crypto/cmh/cmh_aes.c new file +755 -0
755 lines not on this page · raw message
drivers/crypto/cmh/cmh_aes_aead.c new file +1013 -0
1,013 lines not on this page · raw message
drivers/crypto/cmh/cmh_aes_cmac.c new file +476 -0
476 lines not on this page · raw message
drivers/crypto/cmh/cmh_main.c +25 -0
@@ -32,6 +32,7 @@#include "cmh_mqi.h" #include "cmh_txn.h" #include "cmh_rh.h" +#include "cmh_aes.h" #include "cmh_registers.h" #include "cmh_debugfs.h" #include "cmh_sysfs.h"
@@ -227,10 +228,31 @@ static int cmh_probe(struct platform_device *pdev)if (ret) goto err_rh_init; +/* Register AES skcipher algorithms */ + ret =cmh_aes_register (); +if (ret) + goto err_aes_register; + +/* Register AES AEAD algorithms (GCM, CCM) */ + ret =cmh_aes_aead_register (); +if (ret) + goto err_aes_aead_register; + +/* Register AES CMAC algorithm */ + ret =cmh_aes_cmac_register (); +if (ret) + goto err_aes_cmac_register; +platform_set_drvdata (pdev, dev);return 0 ; +err_aes_cmac_register: +cmh_aes_aead_unregister (); +err_aes_aead_register: +cmh_aes_unregister (); +err_aes_register: +cmh_rh_cleanup (cfg); err_rh_init:cmh_tm_cleanup (); err_tm_init:
@@ -255,6 +277,9 @@ static void cmh_remove(struct platform_device *pdev) cfg = &dev->config; +cmh_aes_cmac_unregister (); +cmh_aes_aead_unregister (); +cmh_aes_unregister ();cmh_rh_cleanup (cfg);cmh_tm_cleanup ();cmh_mqi_cleanup (cfg);
drivers/crypto/cmh/include/cmh_aes.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2026 Cryptography Research, Inc. (CRI). +* CMH LKM -- AES Crypto API Drivers +* +* Registers AES algorithms with the Linux crypto subsystem: +* skcipher: ecb/cbc/ctr/cfb/xts(aes) +* aead: gcm/ccm(aes) +* shash: cmac(aes) +*/ + +#ifndef CMH_AES_H +#define CMH_AES_H + +int cmh_aes_register (void ); +void cmh_aes_unregister (void ); + +int cmh_aes_aead_register (void ); +void cmh_aes_aead_unregister (void ); + +int cmh_aes_cmac_register (void ); +void cmh_aes_cmac_unregister (void ); + +#endif /* CMH_AES_H */
Signature · 1 line
2.43.7
[email protected]
AI reviewRe: [PATCH v7 4/5] crypto: cmh - add AES skcipher/aead/cmac
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] crypto: cmh: stack overflow via missing tfm->fb in cmh_aes_cmac_init_tfm() - [High] crypto…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] crypto: cmh: stack overflow via missing tfm->fb in cmh_aes_cmac_init_tfm()
- [High] crypto: cmh: spurious -EINPROGRESS in cmh_aes_complete() for CTR chunk 2
- [Medium] crypto: cmh - missing CCM payload length check for L=2 in cmh_aes_aead_crypt() --
commit c8ce92a98ef076c53f07cde35dc9d8f4e53da1f5
Author: Alex Ousherovitch <[email protected]>
crypto: cmh - add AES skcipher/aead/cmac
This commit registers AES skcipher (ECB, CBC, CTR, CFB, XTS), AEAD (GCM, CCM), and ahash (CMAC) algorithms using the CMH AES core, along with software fallbacks for unsupported hardware corner cases and incremental CMAC operations.
Quoted 5 lines
drivers/crypto/cmh/cmh_aes.c new file +0
-0
[ ... ]
Quoted 66 lines
drivers/crypto/cmh/cmh_aes.c new file +66
-0+static int cmh_aes_ctr_submit_chunk2 (struct skcipher_request *req) +{ +struct crypto_skcipher *tfm =crypto_skcipher_reqtfm (req); +struct cmh_aes_tfm_ctx *tctx =crypto_skcipher_ctx (tfm); +struct cmh_aes_reqctx *rctx =skcipher_request_ctx (req); +struct vcq_cmd cmds[CMH_AES_MAX_PAYLOAD ]; +u32 chunk1 = rctx->ctr_chunk1_len; +u32 chunk2 = rctx->cryptlen - chunk1; +u64 key_ref; +u32 keylen; +u32 idx =0 ; + +/* Clear split flag so next completion is final */ + rctx->ctr_chunk1_len =0 ; + +vcq_add_sys_write (&cmds[idx++],SYS_REF_TEMP , + (u64 )rctx->key_dma,SYS_REF_NONE , + tctx->key.raw.len, + tctx->key.raw.sys_type); + key_ref =SYS_REF_TEMP ; + keylen = tctx->key.raw.len; + +vcq_add_aes_init (&cmds[idx++], rctx->core_id, key_ref, + (u64 )rctx->iv2_dma, keylen, rctx->ivsize, + rctx->aes_mode, rctx->aes_op,0 ); +vcq_add_aes_final (&cmds[idx++], rctx->core_id, + (u64)(rctx->in_dma + chunk1), + (u64)(rctx->out_dma + chunk1), chunk2); +vcq_add_flush (&cmds[idx++], rctx->core_id); + +return cmh_vcq_pack_and_submit_async (cmds, idx, rctx->packed, +CMH_AES_MAX_PACKED , + rctx->target_mbx, + cmh_aes_complete, req, + !!(req->base.flags & +CRYPTO_TFM_REQ_MAY_BACKLOG ), +cmh_tm_async_timeout_jiffies ()); +} + +/* +* Async completion callback -- fires from RH threaded IRQ context. +* +* Unmaps DMA buffers, copies output to req->dst scatterlist, +* updates the IV state, frees temporaries, and completes the request. +* +* For CTR counter-wrap splits, the first chunk completion chains +* into a second VCQ submission rather than finalizing immediately. +*/ +static void cmh_aes_complete (void *data,int error) +{ +struct skcipher_request *req = data; +struct cmh_aes_reqctx *rctx =skcipher_request_ctx (req); + +if (error == -EINPROGRESS ) { +cmh_complete (&req->base, error); +return ; + } + +/* +* CTR counter-wrap: first chunk completed, submit second. +* DMA mappings remain valid (they cover the full buffer). +* +* Recursion depth bounded: chunk2 clears ctr_chunk1_len before +* submission, so the second cmh_aes_complete invocation sees 0 +* and finalizes (max depth = 2). +*/
drivers/crypto/cmh/cmh_aes.c line 396
Quoted 10 lines
drivers/crypto/cmh/cmh_aes.c new file +10
-0+if (rctx->ctr_chunk1_len && !error) { +int ret; + + ret =cmh_aes_ctr_submit_chunk2 (req); + +if (!ret || ret == -EBUSY ) +return ; +/* Submission failed; clean up below */ + error = ret; + }
[Severity: High]
Can this deliver a spurious or duplicate -EINPROGRESS notification to the outer skcipher_request caller when CTR chunk 2 is backlogged?
When an AES-CTR request with CRYPTO_TFM_REQ_MAY_BACKLOG crosses the 64-bit counter wrap boundary in cmh_aes_crypt(), chunk 1 is submitted first. At that point, either cmh_aes_crypt() returned -EINPROGRESS directly, or chunk 1 was backlogged (-EBUSY) and already triggered the single allowed -EINPROGRESS callback when cmh_tm_thread() promoted it.
When chunk 1 finishes in hardware, cmh_aes_complete() calls cmh_aes_ctr_submit_chunk2(), which submits chunk 2 with backlog_ok enabled and cmh_aes_complete as the callback. If tm.cmq is full when chunk 2 is submitted, cmh_tm_post_command() places chunk 2 on tm.backlog and returns -EBUSY.
When cmh_tm_thread() later promotes chunk 2 from tm.backlog to tm.cmq:
drivers/crypto/cmh/cmh_txn.c:cmh_tm_thread() {
...
if (bl_promoted && bl_promoted->complete)
bl_promoted->complete(bl_promoted->completion_data,
-EINPROGRESS);
...
}cmh_aes_complete() receives -EINPROGRESS for chunk 2 and forwards it to cmh_complete(&req->base, -EINPROGRESS), invoking req->base.complete() with -EINPROGRESS even though the initial submission returned -EINPROGRESS (or already fired -EINPROGRESS once for chunk 1).
Quoted 5 lines
drivers/crypto/cmh/cmh_aes_aead.c new file +0
-0
[ ... ]
Quoted 2 lines
drivers/crypto/cmh/cmh_aes_aead.c new file +2
-0+static int cmh_aes_aead_crypt (struct aead_request *req,u32 aes_op) +{
[ ... ]
Quoted 33 lines
drivers/crypto/cmh/cmh_aes_aead.c new file +33
-0+if (aes_op ==AES_OP_ENCRYPT ) { + cryptlen = req->cryptlen; + }else { +if (req->cryptlen < authsize) +return -EINVAL ; + cryptlen = req->cryptlen - authsize; + } + +/* +* Validate CCM IV format early -- the empty-input fallback and +* nonce extraction both depend on iv[0] being in range [1,7]. +*/ +if (info->type ==CMH_AES_AEAD_CCM ) { +if (req->iv[0 ] <1 || req->iv[0 ] >7 ) +return -EINVAL ; + } + +/* +* The CMH eSW rejects GCM/CCM when both aadlen and iolen are zero. +* For GCM, the tag is simply E(K, J0) -- handle with ECB fallback. +* For CCM, compute tag = E(K,B0) XOR E(K,A0) in software. +*/ +if (cryptlen ==0 && req->assoclen ==0 ) { +if (info->type ==CMH_AES_AEAD_GCM ) +return cmh_aes_gcm_empty (req, aes_op); +return cmh_aes_ccm_empty (req, aes_op); + } + +/* +* HW does not support authsize=10 for CCM. Forward the entire +* request to the generic CCM implementation. +*/ +if (info->type ==CMH_AES_AEAD_CCM && authsize ==10 )
drivers/crypto/cmh/cmh_aes_aead.c line 712
Quoted 10 lines
drivers/crypto/cmh/cmh_aes_aead.c new file +10
-0+return cmh_aes_ccm_fallback (req, aes_op); + +/* +* HW uses a proprietary LLI scatter-gather format that is +* incompatible with struct scatterlist, so the payload is +* linearised into contiguous buffers for DMA. Cap total +* size to prevent excessive memory consumption. +*/ +if ((u64 )cryptlen + req->assoclen >SZ_1M ) +return -EINVAL ;
[Severity: Medium]
Should cmh_aes_aead_crypt() check that cryptlen fits within L = req->iv[0] + 1 bytes for CCM, returning -EOVERFLOW when req->iv[0] == 1 and cryptlen > 0xffff?
Per RFC 3610 section 2.1, when req->iv[0] == 1 (L = 2, 13-byte nonce), the maximum representable payload length is 65535 bytes (as checked in crypto/ccm.c:set_msg_len() and lib/crypto/aes.c:aes_ccm_init()). Because cmh_aes_aead_crypt() allows cryptlen up to SZ_1M without checking whether it exceeds 0xffff for L = 2, an oversized L = 2 request is sent to the hardware instead of failing with -EOVERFLOW.
If the eSW rejects the command, cmh_aes_aead_complete() returns -EIO on encrypt and maps -EIO to -EBADMSG on decrypt instead of -EOVERFLOW; on 64K page systems with CONFIG_CRYPTO_MANAGER_EXTRA_TESTS enabled, test_aead_vs_generic_impl() generates vectors above 64 KiB and expects -EOVERFLOW to match the generic CCM implementation.
Quoted 5 lines
drivers/crypto/cmh/cmh_aes_cmac.c new file +0
-0
[ ... ]
Quoted 11 lines
drivers/crypto/cmh/cmh_aes_cmac.c new file +11
-0+/* +* .digest() -- compute the CMAC over the whole message in a single +* hardware transaction: +* SYS_CMD_WRITE(key) + AES_CMD_INIT(CMAC) + AES_CMD_AAD_FINAL_AUTH + +* FLUSH. +* The message is linearised into one DMA-mapped buffer. The crypto core +* routes virtual-address requests to the software fallback before this +* op runs, so req->src is an ordinary scatterlist here. +*/ +static int cmh_aes_cmac_digest (struct ahash_request *req) +{
[ ... ]
Quoted 11 lines
drivers/crypto/cmh/cmh_aes_cmac.c new file +11
-0+if (data_len) { + rctx->buf =kmalloc (data_len, gfp | __GFP_NOWARN); +if (!rctx->buf) +return -ENOMEM ; +if (req->base.flags &CRYPTO_AHASH_REQ_VIRT ) +memcpy (rctx->buf, req->svirt, data_len); +else +scatterwalk_map_and_copy (rctx->buf, req->src,0 , + data_len,0 ); + rctx->data_len = data_len; + }
[ ... ]
Quoted 18 lines
drivers/crypto/cmh/cmh_aes_cmac.c new file +18
-0+static int cmh_aes_cmac_init_tfm (struct crypto_ahash *tfm) +{ +struct cmh_aes_cmac_tfm_ctx *tctx =crypto_ahash_ctx (tfm); +struct crypto_ahash *fb; + + tctx->key.mode =CMH_KEY_NONE ; + +/* +* The generic cmac(aes) shash has no export_core, so the crypto +* core cannot wrap it as crypto_ahash_fb(). Allocate it here +* instead; masking out CRYPTO_ALG_ASYNC excludes this (async) +* driver so the allocator picks the generic, synchronous transform, +* and its request is embedded after the reqctx. +*/ + fb =crypto_alloc_ahash (crypto_ahash_alg_name (tfm),0 , +CRYPTO_ALG_ASYNC ); +if (IS_ERR (fb)) +return PTR_ERR (fb);
drivers/crypto/cmh/cmh_aes_cmac.c line 390
Quoted 10 lines
drivers/crypto/cmh/cmh_aes_cmac.c new file +10
-0+ tctx->fb = fb; + +crypto_ahash_set_reqsize (tfm, +sizeof (struct cmh_aes_cmac_reqctx ) + +crypto_tfm_ctx_alignment () + +sizeof (struct ahash_request ) + +crypto_ahash_reqsize (fb)); +crypto_ahash_set_statesize (tfm,crypto_ahash_statesize (fb)); +return 0 ; +}
[Severity: Critical]
Does leaving tfm->fb pointing to the asynchronous CMH transform in cmh_aes_cmac_init_tfm() overflow the stack buffer in crypto_hash_digest()?
In crypto_create_tfm_node(), tfm->fb is initialized to point to tfm itself. Because cmh_aes_cmac_alg sets CRYPTO_ALG_NO_FALLBACK, crypto_ahash_init_tfm() does not allocate a fallback transform or update tfm->fb, and cmh_aes_cmac_init_tfm() stores the fallback only in tctx->fb while setting tfm's reqsize to:
sizeof(struct cmh_aes_cmac_reqctx) + crypto_tfm_ctx_alignment() + sizeof(struct ahash_request) + crypto_ahash_reqsize(fb)
Since crypto_ahash_init_tfm() raises crypto_ahash_reqsize(fb) to at least HASH_MAX_DESCSIZE, and MAX_SYNC_HASH_REQSIZE is sizeof(struct ahash_request) + HASH_MAX_DESCSIZE, crypto_ahash_reqsize(tfm) exceeds MAX_SYNC_HASH_REQSIZE by sizeof(struct cmh_aes_cmac_reqctx) + crypto_tfm_ctx_alignment() (over 560 bytes).
When a caller invokes crypto_hash_digest(tfm, data, len, out) (or uses HASH_FBREQ_ON_STACK() or HASH_REQUEST_CLONE() when falling back on allocation failure):
crypto/ahash.c:crypto_hash_digest() {
HASH_REQUEST_ON_STACK(req, crypto_ahash_fb(tfm));
int err;
ahash_request_set_callback(req, 0, NULL, NULL);
ahash_request_set_virt(req, data, out, len);
err = crypto_ahash_digest(req);
ahash_request_zero(req);
return err;
}HASH_REQUEST_ON_STACK() allocates __req_req on the stack with size sizeof(struct ahash_request) + MAX_SYNC_HASH_REQSIZE and binds req to crypto_ahash_fb(tfm), which is still the asynchronous CMH tfm. crypto_ahash_digest(req) then fails with -EAGAIN (because ahash_req_on_stack(req) && ahash_is_async(tfm) is true), and ahash_request_zero(req) zeroes sizeof(*req) + crypto_ahash_reqsize(tfm) bytes, overwriting more than 560 bytes past the end of __req_req on the kernel stack.
Also, the commit message states that virtual-address (CRYPTO_AHASH_REQ_VIRT) paths are forwarded to the generic cmac(aes) transform, and the comment above cmh_aes_cmac_digest() notes that the crypto core routes virtual-address requests to the software fallback before .digest() runs. However, because cmh_aes_cmac_alg sets CRYPTO_ALG_REQ_VIRT, ahash_do_req_chain() calls cmh_aes_cmac_digest() directly for CRYPTO_AHASH_REQ_VIRT requests, which copies req->svirt and submits the request asynchronously to hardware instead of forwarding to tctx->fb.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v7 5/5 MAINTAINERS: add Rambus CryptoManager Hub (CMH)
In reply to Alex OusherovitchAdd MAINTAINERS entry for the Rambus CryptoManager Hub (CMH) hardware crypto accelerator driver under drivers/crypto/cmh/.
--- MAINTAINERS | 13 +++++++++++++ 1 file changed, 13 insertions(+)
MAINTAINERS +13 -0
@@ -22760,6 +22760,19 @@ L: [email protected] S: Maintained F: drivers/net/wireless/ralink/ +RAMBUS CRYPTOMANAGER HUB (CMH) HARDWARE CRYPTO ACCELERATOR +M: Alex Ousherovitch <[email protected]> +M: Saravanakrishnan Krishnamoorthy <[email protected]> +R: Joel Wittenauer <[email protected]> +R: Thi Nguyen <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/ABI/testing/debugfs-driver-cmh +F: Documentation/ABI/testing/sysfs-driver-cmh +F: Documentation/crypto/device_drivers/cmh.rst +F: Documentation/devicetree/bindings/crypto/rambus,cmh-v1030.yaml +F: drivers/crypto/cmh/ + RAMDISK RAM BLOCK DEVICE DRIVER M: Jens Axboe <[email protected]> S: Maintained
Signature · 1 line
2.43.7