crypto: hisilicon/hpre: RSA request never completes during signed module verification on Kunpeng arm64

Hello,

I observed a permanently stuck RSA request in the HiSilicon HPRE driver while a signed kernel module was being verified on a Kunpeng ARM64 server. The lost completion subsequently blocked module loading and all PVE API workers which invoked ss(8).

This was observed with a Proxmox kernel rather than a vanilla upstream kernel. I have not yet reproduced it on a mainline build, but the relevant HPRE and rsassa-pkcs1 paths appear unchanged in current upstream source.

Environment:

architecture: arm64
CPU: TaiShan-v120 (Kunpeng)
system product: J353 G3
HPRE PCI ID: 19e5:a258 (four PFs)
kernel: 7.0.14-6-pve
kernel package: proxmox-kernel-7.0.14-6-pve-signed 7.0.14-6 CONFIG_MODULE_SIG=y
Secure Boot: disabled

Trigger and impact:

PVE executed:

/bin/ss -Htln sport = 5900

This requested net-pf-16-proto-4-type-2, which resolves to inet_diag. Verification of the signed inet_diag module selected HPRE because the registered algorithms were:

pkcs1(hpre-rsa,sha512) priority 1000
hpre-rsa priority 1000
rsa-generic priority 100

The first modprobe has remained in uninterruptible sleep for more than ten hours. Its stack is:

  rsassa_pkcs1_verify+0x284/0x3a0
  public_key_verify_signature+0x1e4/0x2b0
  public_key_verify_signature_2+0x20/0x48
  verify_signature+0x48/0xc0
  pkcs7_validate_trust+0x23c/0x300
  verify_pkcs7_message_sig+0x10c/0x190
  verify_pkcs7_signature+0x78/0xe0
  mod_verify_sig+0xac/0x150
  module_sig_check+0x104/0x190
  load_module+0x48/0x2a68
  init_module_from_file+0x110/0x168
  __arm64_sys_finit_module+0x220/0x398

All later ss processes became blocked in call_usermodehelper_exec() while waiting for the same module request. This exhausted all three pvedaemon workers and caused PVE API login requests to time out with HTTP 596.

HPRE debugfs counters isolate the lost completion to one PF:

0000:3a:00.0 send=3 recv=3
0000:3e:00.0 send=25 recv=25
0000:7a:00.0 send=1 recv=0
0000:7e:00.0 send=1 recv=1

For 0000:7a:00.0, send_busy_cnt, send_fail_cnt, invalid_req_cnt, abnormal_irq, aeq_irq, err_irq and dev_timeout are all zero. The QM status still reports "work". Its completion IRQ vector has received zero interrupts; the assigned CPU is online.

The signed kernel package passes dpkg verification. hisi_hpre is not in the initramfs and is loaded automatically through the PCI modalias. It is not exposed through UACCE and is not passed through to a VM.

Possible cause:

hpre_hw_data_init() directly maps a single full-size RSA scatterlist with dma_map_single() through hpre_get_data_dma_addr(), without checking the DMA/cacheline alignment. rsassa_pkcs1_verify() uses the same scatterlist as both source and destination for the child akcipher request.

HPRE already has hpre_prepare_dma_buf(), which allocates a coherent bounce buffer, but the full-size RSA path bypasses it. This resembles the recent rsassa-pkcs1 alignment discussion, where it was noted that the underlying accelerator driver must handle unaligned buffers:

https://lore.kernel.org/linux-crypto/[email protected]/

I cannot prove from the current stuck system that alignment is the only cause, so this is a hypothesis rather than a conclusion. Independently, hpre_is_bd_timeout() only records elapsed time after a response callback; it cannot recover a request whose completion never arrives. The synchronous crypto waiter therefore remains blocked indefinitely.

The production workaround is to blacklist hisi_hpre and reboot, which causes signed-module verification to use rsa-generic. The current request cannot be killed or recovered without rebooting.

Would forcing unaligned or in-place RSA requests through the existing coherent bounce-buffer path be an appropriate first fix? Is there an existing HPRE/QM recovery mechanism that can safely time out the request, reset the affected queue, and complete or fall back to software without a late-callback use-after-free risk?

The corresponding Proxmox report, including a sanitized diagnostic attachment, is available here:

https://bugzilla.proxmox.com/show_bug.cgi?id=7980

Please let me know if a mainline reproduction, additional HPRE registers, or testing of a candidate patch would be useful. The affected production host is currently being kept unchanged so that more read-only evidence can still be collected before the maintenance reboot.

Regards,
Evanshenf
GitHub: https://github.com/Evanshenf

Hello!

We have received your report and have initiated internal analysis. Since we do not have access to your Proxmox kernel, we are using the open-source kernel 7.0.14 for analysis and testing. If you suspect this is an alignment issue, you could try modifying and verifying it.

Thanks.

Hello Longfang and all,

Thank you for the suggestion. I tested the alignment hypothesis on the affected system with an instrumented HPRE module built against the exact 7.0.14-6-pve headers.

The diagnostic patch detects in-place or unaligned RSA requests and routes source and destination through separate dma_alloc_coherent() buffers. Per-PF counters prove that this path was used.

Results:

0000:3e:00.0: send=250, recv=250, bounce=250 0000:3a:00.0: send=250, recv=250, bounce=250 0000:7e:00.0: send=250, recv=250, bounce=250 0000:7a:00.0: first request hung, send=1, recv=0, bounce=1

For the failing PF:

SEND_SQE_TO_ACC=1, ACC_FINISH=1, CQE_OUT=1, EQE_OUT=1 Linux completion IRQ count=0, HPRE recv_cnt=0

The task remained blocked in rsassa_pkcs1_verify(), at the same point as the original incident. Thus, in-place and unaligned buffers are present, but separate coherent bounce buffers do not fix the deterministic completion loss on 0000:7a:00.0. Alignment is not a sufficient root cause. The new evidence narrows the fault to the completion-notification path after accelerator completion and before the HPRE response callback.

Could you please help clarify:

1. What are the exact meanings of QM_DFX_ACC_FINISH_CNT, QM_CQE_OUT, and QM_EQE_OUT?
2. With ACC_FINISH=1, CQE_OUT=1, EQE_OUT=1, and Linux IRQ=0, which additional QM/HPRE/GIC ITS registers would be most useful?
3. Is there a known revision-30 19e5:a258 issue affecting one PF/NUMA node or its MSI path?
4. Is there a safe way to poll/recover the affected CQ/QP and complete the request with an error while preventing a late-callback use-after-free?

I attached the complete report, experiment patch, and a sanitized evidence archive containing four-PF counters, QM/PCI/MSI state, raw IRQ counts, and the blocked-task stack. The archive contains no management addresses, credentials, serial numbers, or unrelated logs.

Attachment SHA-256:

report: 610f08baf93177b0069a406c55279ed73fc1e9cd417ce64c813d8e29fa0e2108
patch: 811332a88a7338cc144b60268d9e9be4bae233ce6a0fdab587bead29ceacec50
data: 42567764d09204758b0f5e01c9b4d93fab4ed0d2a1a15afef64803d8b7c74815

The host is back on the safe baseline with hisi_hpre blacklisted and rsa-generic selected. A 200-cycle signed-module verification test across all four NUMA nodes passed after recovery.

Regards,
Evanshenf
GitHub: https://github.com/Evanshenf

# HPRE RSA Completion-Loss Validation Report

Date: 2026-08-27
Reporter: Evanshenf <[email protected]>
Related report: [Proxmox Bug 7980](https://bugzilla.proxmox.com/show_bug.cgi?id=7980)

## 1. Executive summary

I tested the alignment/in-place hypothesis from the original HPRE RSA completion-loss report by building an instrumented hisi_hpre module against the exact headers of the affected PVE kernel. The diagnostic module forces risky RSA requests through two independent coherent DMA bounce buffers and exposes per-PF counters proving which path was taken.

The experiment produced the following result:

- The RSASSA child request was both in-place and unaligned in every tested module-signature verification.
- The diagnostic path definitely replaced the original source and destination mapping with two independent dma_alloc_coherent() buffers.
- Three HPRE PFs completed 250 signed-module verification cycles each.
- PF 0000:7a:00.0 hung on its first request, exactly as in the original incident.
- On the failing PF, QM reported that the request reached the accelerator and that accelerator completion, CQE_OUT, and EQE_OUT had each advanced by one. However, the Linux completion MSI counter and the HPRE driver recv_cnt both remained zero.

Therefore, buffer alignment and in-place operation are real conditions, but a coherent bounce buffer is not sufficient to fix this failure. The new evidence narrows the fault to the completion notification path after accelerator completion and before the HPRE response callback. Additional QM/HPRE/GIC ITS interpretation from the hardware/driver maintainers is needed to distinguish the exact boundary.

## 2. Test environment

| Item | Value |
| --- | --- |
| Architecture | arm64 |
| CPU | TaiShan-v120 / Kunpeng 920, two sockets, 320 logical CPUs |
| NUMA topology | Four nodes, 80 logical CPUs per node |
| OS | Debian 13 with PVE 9.2.9 |
| Kernel | `7.0.14-6-pve` |
| Kernel package | `proxmox-kernel-7.0.14-6-pve-signed 7.0.14-6` |
| Kernel compiler | GCC 14.2.0 |
| Diagnostic module compiler | GCC 14.3.0 |
| HPRE device | `19e5:a258`, revision `30`, four PFs |
| IOMMU | Translated domain, strict TLB invalidation |
| Test payload | PVE-signed `xfs.ko`; no XFS filesystem was mounted |

PF and NUMA mapping:

```text
0000:3e:00.0 -> NUMA 0 -> CPUs   0-79
0000:3a:00.0 -> NUMA 1 -> CPUs  80-159
0000:7e:00.0 -> NUMA 2 -> CPUs 160-239
0000:7a:00.0 -> NUMA 3 -> CPUs 240-319
```

The PVE package metadata identifies an Ubuntu kernel baseline. The HPRE source used for the diagnostic build was taken from tag Ubuntu-7.0.0-28.28i2, peeled commit 660d1f05673c4e02cc3c2c056afc0dc14c0075f0. The installed and rebuilt baseline modules were also compared at the disassembly level for hpre_hw_data_init(), hpre_rsa_enc(), and hpre_rsa_dec(); the relevant data paths match.

## 3. Diagnostic change

This is an experiment patch, not a proposed production fix.

The module parameter rsa_bounce_mode implements:

```text
0 = retain direct DMA and only observe risky requests
1 = use bounce buffers for in-place or unaligned RSA requests
2 = force bounce buffers for every RSA request
```

Mode 1 was used for this test. When the request is in-place or unaligned, the original direct dma_map_single() path is bypassed for both source and destination. The existing hpre_prepare_dma_buf() path allocates separate coherent source and destination buffers. The source is copied into its coherent buffer, the device writes to a different coherent destination buffer, and the destination is copied back only in the normal completion callback.

The following per-PF counters were added:

```text
rsa_bounce_cnt
rsa_inplace_cnt
rsa_unaligned_cnt
rsa_direct_hazard_cnt
```

These counters are important because they prove that the failing request used the intended bounce path rather than silently falling back to the original direct mapping.

## 4. Method

The diagnostic module was loaded transiently. It did not replace the production module under /lib/modules, and the persistent HPRE blacklist was not removed.

For each NUMA node, the test process was pinned to one representative CPU and repeatedly loaded and unloaded the signed XFS module. The planned count was 250 cycles per node:

```text
CPU 0   -> PF 0000:3e:00.0
CPU 80  -> PF 0000:3a:00.0
CPU 160 -> PF 0000:7e:00.0
CPU 240 -> PF 0000:7a:00.0
```

On failure, the process stack, HPRE and QM debugfs state, PCI configuration, MSI counts and affinities, and relevant kernel logs were captured before recovery.

## 5. Results

### 5.1 Driver and bounce counters

| PF | Test CPU | Completed cycles | send | recv | bounce | in-place | unaligned | direct hazard |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `0000:3e:00.0` | 0 | 250 | 250 | 250 | 250 | 250 | 250 | 0 |
| `0000:3a:00.0` | 80 | 250 | 250 | 250 | 250 | 250 | 250 | 0 |
| `0000:7e:00.0` | 160 | 250 | 250 | 250 | 250 | 250 | 250 | 0 |
| `0000:7a:00.0` | 240 | 0; first request hung | 1 | 0 | 1 | 1 | 1 | 0 |

For the failing PF, bounce=1 and direct_hazard=0 prove that the failing request used separate coherent DMA buffers. The failure is therefore reproducible after removing direct mapping, in-place DMA, and original-buffer alignment from the device data path.

### 5.2 Completion-path counters

| PF | SEND_SQE_TO_ACC | ACC_FINISH | CQE_OUT | EQE_OUT | Linux completion IRQ | HPRE recv |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `0000:3e:00.0` | 250 | 250 | 250 | 250 | 250 | 250 |
| `0000:3a:00.0` | 250 | 250 | 250 | 250 | 250 | 250 |
| `0000:7e:00.0` | 250 | 250 | 250 | 250 | 250 | 250 |
| `0000:7a:00.0` | 1 | 1 | 1 | 1 | 0 | 0 |

The failing request was accepted and reached accelerator completion. QM's CQE_OUT and EQE_OUT counters also advanced, but Linux did not receive the completion vector and the HPRE callback was never entered.

For 0000:7a:00.0, all of the following remained zero:

```text
send_busy_cnt
send_fail_cnt
invalid_req_cnt
over_thrhld_cnt
qm_abnormal_irq
qm_aeq_irq
qm_err_irq
qm_mb_err
qm_create_qp_err
qm_dev_timeout
```

QM continued to report:

```text
status     = work
qm_state   = 1
dev_state  = 0
current_q  = 0
```

### 5.3 MSI and PCI state

| PF | Completion IRQ | Initial effective CPU | Observed IRQ count |
| --- | ---: | ---: | ---: |
| `0000:3e:00.0` | 1374 | 31 | 250 |
| `0000:3a:00.0` | 1370 | 149 | 250 |
| `0000:7e:00.0` | 1382 | 186 | 250 |
| `0000:7a:00.0` | 1378 | 265 | 0 |

The failing PF reported MSI enabled with four vectors, no mask bits, and no pending bits:

```text
MSI: Enable+ Count=4/4 Maskable+ 64bit+
Masking: 00000000
Pending: 00000000
```

PCI Status and Device Status did not report correctable, non-fatal, fatal, or unsupported-request errors. The platform ACPI _OSC reports that AER control is not supported by the platform, so the absence of an AER log is not treated as conclusive evidence that the PCIe layer is fault-free.

After the request had hung, IRQ 1378 affinity was moved from its NUMA-node mask, whose effective CPU was 265, directly to CPU 240. The IRQ count remained zero and the waiter did not recover. This shows that there was no pending completion that could be recovered by retargeting, but it does not alone distinguish failure to generate an MSI from loss in the ITS mapping/delivery path.

### 5.4 Blocked task

The first request on CPU 240 entered uninterruptible sleep:

```text
Name:    modprobe
State:   D (disk sleep)
Command: modprobe xfs

rsassa_pkcs1_verify+0x284/0x3a0
public_key_verify_signature+0x1e4/0x2b0
public_key_verify_signature_2+0x20/0x48
verify_signature+0x48/0xc0
pkcs7_validate_trust+0x23c/0x300
verify_pkcs7_message_sig+0x10c/0x190
verify_pkcs7_signature+0x78/0xe0
mod_verify_sig+0xac/0x150
module_sig_check+0x104/0x190
load_module+0x48/0x2a68
init_module_from_file+0x110/0x168
__arm64_sys_finit_module+0x220/0x398
```

This is the same blocking point as the original production incident.

## 6. Conclusions

### 6.1 Alignment hypothesis

The experiment confirms that the original RSASSA request is in-place and unaligned. However, the same failure occurs after the original SG is completely removed from the device DMA path and replaced by separate coherent source and destination buffers. Alignment or in-place DMA is therefore not a sufficient root cause for this incident.

This result does not argue against the generic RSASSA cacheline-alignment fix. It only shows that such a fix does not resolve the deterministic completion loss on PF 0000:7a:00.0 on this system.

### 6.2 Narrowed failure boundary

The observed boundary is:

```text
RSA request
  -> hisi_qp_send()                       success
  -> QM SEND_SQE_TO_ACC                   +1
  -> HPRE/QM ACC_FINISH                   +1
  -> QM CQE_OUT / EQE_OUT                 +1
  -> Linux MSI completion IRQ             +0  <-- failure interval
  -> HPRE response callback / recv_cnt    +0
  -> akcipher_request_complete()          not called
  -> synchronous RSASSA waiter            blocked indefinitely
```

The exact hardware meaning of CQE_OUT and EQE_OUT is needed before assigning the bug to the QM MSI generator, PCIe MSI write, GIC ITS mapping, or the Linux QM interrupt/queue handling path.

### 6.3 Missing recovery behavior

hpre_is_bd_timeout() is only called from the response callback. It records elapsed time after a response has arrived; it is not a request timer and cannot recover a lost completion. If the callback never occurs, the synchronous crypto waiter can remain blocked indefinitely.

This is a separate recoverability issue. Any timeout/reset design must also prevent a late completion from using a freed request, DMA buffer, or crypto request object.

## 7. Recovery verification

Because the request was in uninterruptible sleep, a normal reboot stalled during shutdown. After all evidence had been captured, the host was reset through its management controller.

The safe production baseline remains:

```text
hisi_hpre: not loaded and persistently blacklisted
RSA implementation: pkcs1(rsa-generic,sha512)
```

After recovery, signed XFS module load/unload was repeated 50 times on each of CPUs 0, 80, 160, and 240, for 200 total software-RSA verifications. All 200 passed in 65 seconds. PVE services and the hosted workloads were restored successfully.

## 8. Questions for the HPRE/QM maintainers

1. What are the exact hardware semantics of QM_DFX_ACC_FINISH_CNT, QM_CQE_OUT, and QM_EQE_OUT?
2. With ACC_FINISH=1, CQE_OUT=1, EQE_OUT=1, and Linux completion IRQ count 0, which additional QM, HPRE, or GIC ITS registers should be captured?
3. Is there a known revision-30 19e5:a258 issue affecting one PF/NUMA node or its MSI path?
4. Is there a safe diagnostic method to poll the CQ/EQ and distinguish a written completion with a lost MSI from an incomplete CQ/EQ writeback?
5. Is there an existing or recommended mechanism to time out and reset one QP/PF, complete the request with an error, and fall back to software while safely rejecting a late callback?

## 9. Artifact hashes

| Artifact | SHA-256 |
| --- | --- |
| Diagnostic module | `08325bc5f5085ba661d592804cf4e55aa8d122e5045e00323166595c81eb7d14` |
| Experiment patch | `811332a88a7338cc144b60268d9e9be4bae233ce6a0fdab587bead29ceacec50` |
| Full internal failure archive | `9a91f6545ca1ea84290cd8b29cc74100682d700f8610a978b566c6943f58b74c` |
| Software-RSA recovery log | `7cf8da71d108b4f7384f2923ed32ce39999255d73e32ea8024389d71db536397` |

The external evidence bundle accompanying this report is sanitized and contains only the patch, counter summaries, selected QM/PCI/IRQ state, and the blocked-task stack. It does not contain management addresses, credentials, serial numbers, or unrelated system logs.

diff -ruN /home/arch/pve-hpre-lab/source-ubuntu-7.0.0-28.28i2/hpre/hpre.h /home/arch/pve-hpre-lab/candidate-hpre-bounce/hpre/hpre.h
@@ -40,6 +40,10 @@
 	HPRE_OVER_THRHLD_CNT,
 	HPRE_OVERTIME_THRHLD,
 	HPRE_INVALID_REQ_CNT,
+	HPRE_RSA_BOUNCE_CNT,
+	HPRE_RSA_INPLACE_CNT,
+	HPRE_RSA_UNALIGNED_CNT,
+	HPRE_RSA_DIRECT_HAZARD_CNT,
 	HPRE_DFX_FILE_NUM
 };
 

diff -ruN /home/arch/pve-hpre-lab/source-ubuntu-7.0.0-28.28i2/hpre/hpre_main.c /home/arch/pve-hpre-lab/candidate-hpre-bounce/hpre/hpre_main.c
@@ -350,7 +350,11 @@
 	"send_busy_cnt",
 	"over_thrhld_cnt",
 	"overtime_thrhld",
-	"invalid_req_cnt"
+	"invalid_req_cnt",
+	"rsa_bounce_cnt",
+	"rsa_inplace_cnt",
+	"rsa_unaligned_cnt",
+	"rsa_direct_hazard_cnt"
 };
 
 /* define the HPRE's dfx regs region and region length */

Hello, We have analyzed and pinpointed this issue, and found that the following modification can resolve it. You may give it a try:

index 99fcc5601ea4..fa45fd7ed173 100644
@@ -2665,6 +2665,10 @@ static int its_alloc_tables(struct its_node *its)
 			indirect = its_parse_indirect_baser(its, baser, &order,
 							    ITS_MAX_VPEID_BITS);
 			break;
+		case GITS_BASER_TYPE_COLLECTION:
+			indirect = its_parse_indirect_baser(its, baser, &order,
+							order_base_2(num_possible_cpus()));
+			break;
 		}
 
 		err = its_setup_baser(its, baser, cache, shr, order, indirect);

Thanks.
Longfang.

Hello Longfang and all,

Thank you for analyzing the issue and providing the GIC ITS Collection BASER change. I applied the exact change to the source baseline used by the installed PVE kernel and completed a real-machine validation.

The result is PASS.

The original kernel allocated only 256 flat Interrupt Collection entries on each ITS:

allocated 256 Interrupt Collections (... flat, esz 16, psz 4K ...)

This system has 320 possible CPUs, and the completion IRQ of the failing HPRE PF was effectively targeted at CPU 265. With the proposed change, order_base_2(num_possible_cpus()) caused all four ITS instances to allocate 512 flat Collection entries:

allocated 512 Interrupt Collections (... flat, esz 16, psz 4K ...)

The table remained flat on this hardware; the effective fix was increasing its capacity from 256 to 512 entries.

For deterministic testing, I used a one-time kexec image with the same 7.0.14-6-pve release. The ITS change was the only functional source patch. I built HPRE/QM in and added only the original kernel's public module-signing certificate to the test keyring, so package-signed xfs.ko verification selected:

pkcs1(hpre-rsa,sha512), priority 1000

No private key was used. The production kernel and initramfs were not replaced.

I then explicitly set the completion IRQ for 0000:7a:00.0 to CPU 265 to reproduce the original out-of-range Collection ID and ran signed-module load/unload tests pinned to each NUMA node.

Results:

PF CPU cycles send recv completion IRQ delta

  0000:3e:00.0        0      250    250    250       250
  0000:3a:00.0       80      250    250    250       250
  0000:7e:00.0      160      250    250    250       250

0000:7a:00.0 240 1250 1250 1250 1250

For the original failing PF, the completion vector remained on effective CPU 265 throughout the test. Its IRQ count changed from 2 to 1252, exactly matching the 1250 HPRE completions.

Across all 2000 hardware RSA verifications:

send_busy_cnt = 0
send_fail_cnt = 0
invalid_req_cnt = 0
watchdog events = 0
residual D tasks = 0

There were no HPRE/QM error or timeout messages and no rsassa_pkcs1_verify blocked-task warning.

After validation, I performed a normal firmware reboot. The host returned to the untouched signed production kernel, the HPRE blacklist remained active, PVE services and hosted workloads recovered, and software-RSA module verification passed.

This confirms that the undersized ITS Collection table caused the reproduced completion loss and that your change resolves it on this 320-CPU Kunpeng system.

Please feel free to add:

If you submit this change upstream, I would appreciate a link to the patch thread so I can follow any review or run additional tests if requested.

Regards,
Evanshenf
GitHub: https://github.com/Evanshenf