Partial reconfiguration on ZynqMP
Hi everyone,
We are currently trying to use partial reconfiguration for our ZynqMP- based system (part is xczu6eg) and device-tree overlay based reconfiguration crashes.
Our Linux system is using 5.10.0-xilinx-v2021.1.
The reconfigurable partition is decoupled from the rest of the design with an AXI-shutdown manager IP and a DFX decoupler (the shutdown_requested signal of the AXI shutdown manager is used to drive the request_shutdown of the DFX decoupler).
The relevant parts of the device tree are:
---- BEGIN SNIP ----
fpga_full: fpga-full {
status = "okay";
reconf_region: reconf_region0 {
compatible = "fpga-region";
fpga-bridges = <&reconf_bridge0>;
};
};reconf_bridge0: reconf_bridge@a0080000 {
status = "okay";
compatible = "xlnx,dfx-axi-shutdown-manager-
1.00";
reg = <0x0 0xa0080000 0x0 0x10>;
clock-names = "aclk";
clocks = <&zynqmp_clk 73>;
bridge-enable = <1>;
};---- END SNIP ----
To reconfigure, I tested the following cases:
1. manual reconfiguration: WORKS
- decouple by writing to the register directly
- reconfigure directly with the fpga-manager
- recouple by writing directly to the register
2. full reconfiguration with device-tree overlay: WORKS
- we feed the full bitstream to the FPGA manager
- using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;
&reconf_region {
firmware-name = "full.bit";
};---- BEGIN SNIP ----
3. partial reconfiguration with device-tree overlay:
- we feed the full bitstream to the FPGA manager
- using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;
&reconf_region {
partial-fpga-config;
fpga-bridges = <&reconf_bridge0>;
firmware-name = "partial.bit";
};---- BEGIN SNIP ----
In this case, the whole system crashes (even the platform cable does not respond anymore).
When setting an ILA trigger on the write valid signal of the AXI control path of the AXI Shutdown manager, the ILA appears to trigger but crashes before the data is shown in the hardware manager window of Vivado.
When the partial bitstream written by the overlay is already loaded (by hand, as is case 1), the reconfiguration using an overlay works and the AXI Shutdown manager is written to as expected.
Do you have an idea on what is going wrong here ? As I manage to do the reconfiguration by hand, I suspect a configuration/device-tree/software error.
Best regards
Simon Doppler
A message that is not in the archive (sent off-list or not yet received): BYAPR02MB3925AB4E544B1E93232A4BB2D97D9@BYAPR02MB3925.namprd02.prod.outlook.com
Hi ,
Please find my response inline.
Quoted 75 lines
Quoted 31 lines
-----Original Message-----
From: Simon Doppler <[email protected]>
Sent: Wednesday, December 22, 2021 2:14 PM
To: [email protected]; [email protected]; Michal Simek <[email protected]>; [email protected]
Cc: Adrian Fiergolski <[email protected]>
Subject: Partial reconfiguration on ZynqMPHi everyone,
We are currently trying to use partial reconfiguration for our ZynqMP- based system (part is xczu6eg) and device-tree overlay based reconfiguration crashes.
Our Linux system is using 5.10.0-xilinx-v2021.1.The reconfigurable partition is decoupled from the rest of the design with an AXI-shutdown manager IP and a DFX decoupler (the
shutdown_requestedsignal of the AXI shutdown manager is used to drive therequest_shutdownof the DFX decoupler).The relevant parts of the device tree are:
---- BEGIN SNIP ----
fpga_full: fpga-full { status = "okay"; reconf_region: reconf_region0 { compatible = "fpga-region"; fpga-bridges = <&reconf_bridge0>; }; };Quoted 3 lines
reconf_bridge0: reconf_bridge@a0080000 { status = "okay"; compatible = "xlnx,dfx-axi-shutdown-manager-1.00";
Quoted 40 lines
reg = <0x0 0xa0080000 0x0 0x10>; clock-names = "aclk"; clocks = <&zynqmp_clk 73>; bridge-enable = <1>; };---- END SNIP ----
To reconfigure, I tested the following cases:
1. manual reconfiguration: WORKS
- decouple by writing to the register directly
- reconfigure directly with the fpga-manager
- recouple by writing directly to the register2. full reconfiguration with device-tree overlay: WORKS
- we feed the full bitstream to the FPGA manager
- using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;&reconf_region { firmware-name = "full.bit"; };---- BEGIN SNIP ----
3. partial reconfiguration with device-tree overlay:
- we feed the full bitstream to the FPGA manager
- using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;&reconf_region { partial-fpga-config; fpga-bridges = <&reconf_bridge0>; firmware-name = "partial.bit"; };---- BEGIN SNIP ----
Quoted 15 lines
Quoted 5 lines
In this case, the whole system crashes (even the platform cable does not respond anymore).
When setting an ILA trigger on the write valid signal of the AXI control path of the AXI Shutdown manager, the ILA appears to trigger but crashes before the data is shown in the hardware manager window ofVivado.
Quoted 9 lines
When the partial bitstream written by the overlay is already loaded (by hand, as is case 1), the reconfiguration using an overlay works and the AXI Shutdown manager is written to as expected.
Do you have an idea on what is going wrong here ? As I manage to do the reconfiguration by hand, I suspect a configuration/device-tree/software error.
In the case 2 use case could you please provide more details (if possible please share the log) needs to cross verify whether the axi shutdown manager is written/configured as expected.
Regards,
Navakishore.
Hi,
See my reply below.
Quoted 101 lines · Nava kishore Manne
Hi ,
Please find my response inline.Quoted 86 lines
Quoted 34 lines
-----Original Message-----
From: Simon Doppler <[email protected]>
Sent: Wednesday, December 22, 2021 2:14 PM
To: [email protected]; [email protected]; Michal Simek <[email protected]>; [email protected]
Cc: Adrian Fiergolski <[email protected]>
Subject: Partial reconfiguration on ZynqMPHi everyone,
We are currently trying to use partial reconfiguration for our ZynqMP-
based system (part is xczu6eg) and device-tree overlay based reconfiguration crashes.
Our Linux system is using 5.10.0-xilinx-v2021.1.The reconfigurable partition is decoupled from the rest of the design
with an AXI-shutdown manager IP and a DFX decoupler (theshutdown_requestedsignal of the AXI shutdown manager is used to drive
therequest_shutdownof the DFX decoupler).The relevant parts of the device tree are:
---- BEGIN SNIP ----
fpga_full: fpga-full { status = "okay"; reconf_region: reconf_region0 { compatible = "fpga-region"; fpga-bridges = <&reconf_bridge0>; }; };Quoted 3 lines
reconf_bridge0: reconf_bridge@a0080000 { status = "okay"; compatible = "xlnx,dfx-axi-shutdown-manager-1.00";
Quoted 40 lines
reg = <0x0 0xa0080000 0x0 0x10>; clock-names = "aclk"; clocks = <&zynqmp_clk 73>; bridge-enable = <1>; };---- END SNIP ----
To reconfigure, I tested the following cases:
1. manual reconfiguration: WORKS
- decouple by writing to the register directly
- reconfigure directly with the fpga-manager
- recouple by writing directly to the register2. full reconfiguration with device-tree overlay: WORKS
- we feed the full bitstream to the FPGA manager
- using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;&reconf_region { firmware-name = "full.bit"; };---- BEGIN SNIP ----
3. partial reconfiguration with device-tree overlay:
- we feed the full bitstream to the FPGA manager
- using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;&reconf_region { partial-fpga-config; fpga-bridges = <&reconf_bridge0>; firmware-name = "partial.bit"; };---- BEGIN SNIP ----
Quoted 8 lines
In this case, the whole system crashes (even the platform cable does
not respond anymore).
When setting an ILA trigger on the write valid signal of the AXI control path of the AXI Shutdown manager, the ILA appears to trigger
but crashes before the data is shown in the hardware manager window ofQuoted 11 lines
Vivado.
Quoted 10 lines
When the partial bitstream written by the overlay is already loaded
(by hand, as is case 1), the reconfiguration using an overlay works
and the AXI Shutdown manager is written to as expected.Do you have an idea on what is going wrong here ? As I manage to do the reconfiguration by hand, I suspect a configuration/device-tree/software error.
Quoted 5 lines
In the case 2 use case could you please provide more details (if possible please share the log)
needs to cross verify whether the axi shutdown manager is written/configured as expected.
In case 2, the bridge is not triggered as I replace the full bitstream including the bridge. The relevant parts of the log are below:
U-Boot SPL 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[....]
U-Boot 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[....]
Scanning mmc 1:1...
Found U-Boot script /boot.scr
509 bytes read in 6 ms (82 KiB/s)
## Executing script at 20000000
26510890 bytes read in 1759 ms (14.4 MiB/s)
design filename = "Falcon;UserID=0XFFFFFFFF;Version=2021.1" part number = "xczu6eg-ffvc900-1-i"
date = "2021/12/20"
time = "16:29:50"
bytes in bitstream = 26510780zynqmp_align_dma_buffer: Align buffer at 000000000010006e to 00000000000fff80(swap 0)
49559 bytes read in 16 ms (3 MiB/s)
21019136 bytes read in 1393 ms (14.4 MiB/s)
## Flattened Device Tree blob at 00100000
Booting using the fdt blob at 0x100000
Loading Device Tree to 000000007de03000, end 000000007de12196 ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034] [ 0.000000] Linux version 5.10.0-xilinx-v2021.1 (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1.20210209) #1 SMP Tue Oct 12 09:30:57 UTC 2021 [....] [ 0.000000] Kernel command line: video=DP-1:1920x1080 xilinx- video.is_mplane=0 root=/dev/mmcblk1p2 rw rootwait earlycon clk_ignore_unused [....] [ 3.217384] zynqmp-ipi-mbox mailbox@ff990400: Registered ZynqMP IPI mbox with TX/RX channels. [ 3.217581] FPGA manager framework [....] [ 3.439816] fpga_manager fpga0: Xilinx ZynqMP FPGA Manager registered [ 3.440223] xlnx_pr_decoupler a0080000.user_proc_bridge: fpga bridge
[Xilinx DFX AXI shutdown mgr] registered
[....]
falcon login: root
root@falcon:~# mkdir /sys/kernel/config/device-tree/overlays/user partialBitstream-average-fwonly.dtbo partialBitstream-average- nobridge.dtbo partialBitstream-average- nofw.dtbo partialBitstream-average-nopartial.dtbo root@falcon:~# cat full.dtbo > /sys/kernel/config/device- tree/overlays/user/dtbo [ 92.045900] fpga_manager fpga0: writing partialBitstream-average.bit to Xilinx ZynqMP FPGA Manager [ 104.099858] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/reconf_region0/fpga-bridges [ 104.111882] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/reconf_region0/firmware-name root@falcon:~#
--- END LOG ---
When loading the partial bitstream, I get the following log:
U-Boot SPL 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[... irrelevant U-Boot SPL log ...]
U-Boot 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[... irrelevant U-Boot log ...]
Scanning mmc 1:1...
Found U-Boot script /boot.scr
509 bytes read in 6 ms (82 KiB/s)
## Executing script at 20000000
26510890 bytes read in 1786 ms (14.2 MiB/s)
design filename = "Falcon;UserID=0XFFFFFFFF;Version=2021.1" part number = "xczu6eg-ffvc900-1-i"
date = "2021/12/20"
time = "16:29:50"
bytes in bitstream = 26510780zynqmp_align_dma_buffer: Align buffer at 000000000010006e to 00000000000fff80(swap 0)
49559 bytes read in 16 ms (3 MiB/s)
21019136 bytes read in 1413 ms (14.2 MiB/s)
## Flattened Device Tree blob at 00100000
Booting using the fdt blob at 0x100000
Loading Device Tree to 000000007de03000, end 000000007de12196 ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034] [ 0.000000] Linux version 5.10.0-xilinx-v2021.1 (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1.20210209) #1 SMP Tue Oct 12 09:30:57 UTC 2021 [....] [ 0.000000] Kernel command line: video=DP-1:1920x1080 xilinx- video.is_mplane=0 root=/dev/mmcblk1p2 rw rootwait earlycon clk_ignore_unused [....] [ 2.372821] zynqmp-ipi-mbox mailbox@ff990400: Registered ZynqMP IPI mbox with TX/RX channels. [ 2.373018] FPGA manager framework [....] [ 2.595794] fpga_manager fpga0: Xilinx ZynqMP FPGA Manager registered [ 2.596206] xlnx_pr_decoupler a0080000.user_proc_bridge: fpga bridge
[Xilinx DFX AXI shutdown mgr] registered
[....]
falcon login: root
root@falcon:~# mkdir /sys/kernel/config/device-tree/overlays/user root@falcon:~# file /lib/firmware/partial.bit /lib/firmware/partial.bit: Xilinx BIT data - from Falcon;PARTIAL=TRUE;UserID=0XFFFFFFFF;COMPRESS=FALSE;Version=2021.1 - for xczu6eg-ffvc900-1-i - built 2021/12/20(16:35:18) - data length 0xf4510 root@falcon:~# cat partial.dtbo > /sys/kernel/config/device- tree/overlays/user/dtbo [ 259.672997] fpga_manager fpga0: writing partialBitstream-average.bit to Xilinx ZynqMP FPGA Manager
[ system crash at this point ]
--- END LOG ---
The result of the Vivado Hardware manager is here:
https://nextcloud.bfh.science/index.php/s/HWSEfzawEm2kWpa
As you can see, the System ILA triggerd on a AWVALID signal rising edge with the 0xA0080000 address (the address of the bridge). There does not appear to be a read to that address which would check the the decoupling actually happened.
Is is possible that there is a delay missing when the overlay is applied? This delay is present since I have to type the command when I do it manually (and then it works).
Quoted 2 lines
Regards,
Navakishore.
Best regards
Simon