FPGA DFL updates
Subsystems (3)
Moritz Fischer
maintainercover FPGA DFL updates
Hi Greg,
can you please take the following patches. They're mostly new features and some cleanup of the DFL internals.
They've been on the mailing list and have been reviewed.
Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next
commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst") and fixed up PATCH 05/15 to apply on top of that.
If you prefer the original series against char-misc-next let me know, and I'll resubmit.
Thanks,
Moritz
Wu Hao (15):
fpga: dfl-fme-mgr: fix FME_PR_INTFC_ID register address.
fpga: dfl: fme: remove copy_to_user() in ioctl for PR
fpga: dfl: fme: align PR buffer size per PR datawidth
fpga: dfl: fme: support 512bit data width PR
Documentation: fpga: dfl: add descriptions for virtualization and new
interfaces.
fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
fpga: dfl: pci: enable SRIOV support.
fpga: dfl: afu: add AFU state related sysfs interfaces
fpga: dfl: afu: add userclock sysfs interfaces.
fpga: dfl: add id_table for dfl private feature driver
fpga: dfl: afu: export __port_enable/disable function.
fpga: dfl: afu: add error reporting support.
fpga: dfl: afu: add STP (SignalTap) support
fpga: dfl: fme: add capability sysfs interfaces
fpga: dfl: fme: add global error reporting support
.../ABI/testing/sysfs-platform-dfl-fme | 98 +++++ .../ABI/testing/sysfs-platform-dfl-port | 104 +++++ Documentation/fpga/dfl.rst | 100 +++++ drivers/fpga/Makefile | 3 +- drivers/fpga/dfl-afu-error.c | 225 ++++++++++ drivers/fpga/dfl-afu-main.c | 330 ++++++++++++++- drivers/fpga/dfl-afu.h | 7 + drivers/fpga/dfl-fme-error.c | 385 ++++++++++++++++++ drivers/fpga/dfl-fme-main.c | 120 +++++- drivers/fpga/dfl-fme-mgr.c | 117 +++++- drivers/fpga/dfl-fme-pr.c | 65 +-- drivers/fpga/dfl-fme.h | 7 +- drivers/fpga/dfl-pci.c | 40 ++ drivers/fpga/dfl.c | 169 +++++++- drivers/fpga/dfl.h | 54 ++- include/uapi/linux/fpga-dfl.h | 32 ++ 16 files changed, 1776 insertions(+), 80 deletions(-) create mode 100644 drivers/fpga/dfl-afu-error.c create mode 100644 drivers/fpga/dfl-fme-error.c
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 1/15 fpga: dfl-fme-mgr: fix FME_PR_INTFC_ID register address.
From: Wu Hao <[email protected]>
FME_PR_INTFC_ID is used as compat_id for fpga manager and region, but high 64 bits and low 64 bits of the compat_id are swapped by mistake. This patch fixes this problem by fixing register address.
--- drivers/fpga/dfl-fme-mgr.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/fpga/dfl-fme-mgr.c +2 -2
@@ -30,8 +30,8 @@#define FME_PR_STS 0x10#define FME_PR_DATA 0x18#define FME_PR_ERR 0x20 -#define FME_PR_INTFC_ID_H 0xA8 -#define FME_PR_INTFC_ID_L 0xB0 +#define FME_PR_INTFC_ID_L 0xA8 +#define FME_PR_INTFC_ID_H 0xB0/* FME PR Control Register Bitfield */ #define FME_PR_CTRL_PR_RST BIT_ULL(0)/* Reset PR engine */
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 2/15 fpga: dfl: fme: remove copy_to_user() in ioctl for PR
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch removes copy_to_user() code in partial reconfiguration ioctl, as it's useless as user never needs to read the data structure after ioctl.
--- drivers/fpga/dfl-fme-pr.c | 3 --- 1 file changed, 3 deletions(-)
drivers/fpga/dfl-fme-pr.c +0 -3
@@ -159,9 +159,6 @@ static int fme_pr(struct platform_device *pdev, unsigned long arg)mutex_unlock (&pdata->lock); free_exit:vfree (buf); -if (copy_to_user ((void __user *)arg, &port_pr, minsz)) -return -EFAULT ; -return ret; }
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 3/15 fpga: dfl: fme: align PR buffer size per PR datawidth
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
Current driver checks if input bitstream file size is aligned or not per PR data width (default 32bits). It requires one additional step for end user when they generate the bitstream file, padding extra zeros to bitstream file to align its size per PR data width, but they don't have to as hardware will drop extra padding bytes automatically.
In order to simplify the user steps, this patch aligns PR buffer size per PR data width in driver, to allow user to pass unaligned size bitstream files to driver.
--- drivers/fpga/dfl-fme-pr.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/fpga/dfl-fme-pr.c +9 -5
@@ -74,6 +74,7 @@ static int fme_pr(struct platform_device *pdev, unsigned long arg)struct dfl_fme *fme;unsigned long minsz;void *buf =NULL ; +size_t length;int ret =0 ;u64 v;
@@ -85,9 +86,6 @@ static int fme_pr(struct platform_device *pdev, unsigned long arg)if (port_pr.argsz < minsz || port_pr.flags)return -EINVAL ; -if (!IS_ALIGNED (port_pr.buffer_size,4 )) -return -EINVAL ; -/* get fme header region */ fme_hdr =dfl_get_feature_ioaddr_by_id (&pdev->dev,FME_FEATURE_ID_HEADER );
@@ -103,7 +101,13 @@ static int fme_pr(struct platform_device *pdev, unsigned long arg) port_pr.buffer_size))return -EFAULT ; - buf =vmalloc (port_pr.buffer_size); +/* +* align PR buffer per PR bandwidth, as HW ignores the extra padding +* data automatically. +*/ + length =ALIGN (port_pr.buffer_size,4 ); + + buf =vmalloc (length);if (!buf)return -ENOMEM ;
@@ -140,7 +144,7 @@ static int fme_pr(struct platform_device *pdev, unsigned long arg)fpga_image_info_free (region->info); info->buf = buf; - info->count = port_pr.buffer_size; + info->count = length; info->region_id = port_pr.port_id; region->info = info;
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 4/15 fpga: dfl: fme: support 512bit data width PR
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
In early partial reconfiguration private feature, it only supports 32bit data width when writing data to hardware for PR. 512bit data width PR support is an important optimization for some specific solutions (e.g. XEON with FPGA integrated), it allows driver to use AVX512 instruction to improve the performance of partial reconfiguration. e.g. programming one 100MB bitstream image via this 512bit data width PR hardware only takes ~300ms, but 32bit revision requires ~3s per test result.
Please note now this optimization is only done on revision 2 of this PR private feature which is only used in integrated solution that AVX512 is always supported. This revision 2 hardware doesn't support 32bit PR.
--- drivers/fpga/dfl-fme-main.c | 3 + drivers/fpga/dfl-fme-mgr.c | 113 +++++++++++++++++++++++++++++++----- drivers/fpga/dfl-fme-pr.c | 43 +++++++++----- drivers/fpga/dfl-fme.h | 2 + drivers/fpga/dfl.h | 5 ++ 5 files changed, 135 insertions(+), 31 deletions(-)
drivers/fpga/dfl-fme-main.c +3 -0
@@ -21,6 +21,8 @@#include "dfl.h" #include "dfl-fme.h" +#define DRV_VERSION "0.8" +static ssize_t ports_num_show (struct device *dev,struct device_attribute *attr,char *buf) {
@@ -277,3 +279,4 @@ MODULE_DESCRIPTION("FPGA Management Engine driver");MODULE_AUTHOR ("Intel Corporation" );MODULE_LICENSE ("GPL v2" );MODULE_ALIAS ("platform:dfl-fme" ); +MODULE_VERSION (DRV_VERSION );
drivers/fpga/dfl-fme-mgr.c +97 -16
178 lines not on this page · raw message
drivers/fpga/dfl-fme-pr.c +28 -15
84 lines not on this page · raw message
drivers/fpga/dfl-fme.h +2 -0
@@ -21,12 +21,14 @@/** * struct dfl_fme - dfl fme private data * +* @pr_datawidth: data width for partial reconfiguration. * @mgr: FME's FPGA manager platform device. * @region_list: linked list of FME's FPGA regions. * @bridge_list: linked list of FME's FPGA bridges. * @pdata: fme platform device's pdata. */ struct dfl_fme { +int pr_datawidth;struct platform_device *mgr;struct list_head region_list;struct list_head bridge_list;
drivers/fpga/dfl.h +5 -0
@@ -331,6 +331,11 @@ static inline bool dfl_feature_is_port(void __iomem *base) (FIELD_GET (DFH_ID , v) ==DFH_ID_FIU_PORT ); } +static inline u8 dfl_feature_revision (void __iomem *base) +{ +return (u8 )FIELD_GET (DFH_REVISION ,readq (base +DFH )); +} +/** * struct dfl_fpga_enum_info - DFL FPGA enumeration information *
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 5/15 Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)
Documentation/fpga/dfl.rst +100 -0
@@ -87,6 +87,8 @@ The following functions are exposed through ioctls: - Get driver API version (DFL_FPGA_GET_API_VERSION) - Check for extensions (DFL_FPGA_CHECK_EXTENSION) - Program bitstream (DFL_FPGA_FME_PORT_PR) +- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN) +- Release port from PF (DFL_FPGA_FME_PORT_RELEASE) More functions are exposed through sysfs (/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -143,6 +145,9 @@ More functions are exposed through sysfs: Read Accelerator GUID (afu_id) afu_id indicates which PR bitstream is programmed to this AFU. + Global error reporting management (errors/) + error reporting sysfs interfaces allow user to read errors detected by the + hardware, and clear the logged errors. DFL Framework Overview ======================
@@ -218,6 +223,101 @@ the compat_id exposed by the target FPGA region. This check is usually done by userspace before calling the reconfiguration IOCTL. +FPGA virtualization - PCIe SRIOV +================================ +This section describes the virtualization support on DFL based FPGA device to +enable accessing an accelerator from applications running in a virtual machine +(VM). This section only describes the PCIe based FPGA device with SRIOV support. + +Features supported by the particular FPGA device are exposed through Device +Feature Lists, as illustrated below: + +:: + + +-------------------------------+ +-------------+ + | PF | | VF | + +-------------------------------+ +-------------+ + ^ ^ ^ ^ + | | | | + +-----|------------|---------|--------------|-------+ + | | | | | | + | +-----+ +-------+ +-------+ +-------+ | + | | FME | | Port0 | | Port1 | | Port2 | | + | +-----+ +-------+ +-------+ +-------+ | + | ^ ^ ^ | + | | | | | + | +-------+ +------+ +-------+ | + | | AFU | | AFU | | AFU | | + | +-------+ +------+ +-------+ | + | | + | DFL based FPGA PCIe Device | + +---------------------------------------------------+ + +FME is always accessed through the physical function (PF). + +Ports (and related AFUs) are accessed via PF by default, but could be exposed +through virtual function (VF) devices via PCIe SRIOV. Each VF only contains +1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators) +created via PCIe SRIOV interface, to virtual machines. + +The driver organization in virtualization case is illustrated below: +:: + + +-------++------++------+ | + | FME || FME || FME | | + | FPGA || FPGA || FPGA | | + |Manager||Bridge||Region| | + +-------++------++------+ | + +-----------------------+ +--------+ | +--------+ + | FME | | AFU | | | AFU | + | Module | | Module | | | Module | + +-----------------------+ +--------+ | +--------+ + +-----------------------+ | +-----------------------+ + | FPGA Container Device | | | FPGA Container Device | + | (FPGA Base Region) | | | (FPGA Base Region) | + +-----------------------+ | +-----------------------+ + +------------------+ | +------------------+ + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | + +------------------+ Host | Machine +------------------+ + -------------------------------------- | ------------------------------ + +---------------+ | +---------------+ + | PCI PF Device | | | PCI VF Device | + +---------------+ | +---------------+ + +FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device +is detected. It: + +* Finishes enumeration on both FPGA PCIe PF and VF device using common + interfaces from DFL framework. +* Supports SRIOV. + +The FME device driver plays a management role in this driver architecture, it +provides ioctls to release Port from PF and assign Port to PF. After release +a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV +sysfs interface. + +To enable accessing an accelerator from applications running in a VM, the +respective AFU's port needs to be assigned to a VF using the following steps: + +#. The PF owns all AFU ports by default. Any port that needs to be + reassigned to a VF must first be released through the + DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device. + +#. Once N ports are released from PF, then user can use command below + to enable SRIOV and VFs. Each VF owns only one Port with AFU. + + :: + + echo N > $PCI_DEVICE_PATH/sriov_numvfs + +#. Pass through the VFs to VMs + +#. The AFU under VF is accessible from applications in VM (using the + same driver inside the VF). + +Note that an FME can't be assigned to a VF, thus PR and other management +functions are only available via the PF. + Device enumeration ================== This section introduces how applications enumerate the fpga device from
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 6/15 fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this PORT_RELEASE ioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke this PORT_ASSIGN ioctl to attach the port back to PF.
Ioctl interfaces:
* DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.
* DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.
--- drivers/fpga/dfl-fme-main.c | 54 +++++++++++++++++ drivers/fpga/dfl.c | 107 ++++++++++++++++++++++++++++++---- drivers/fpga/dfl.h | 10 ++++ include/uapi/linux/fpga-dfl.h | 32 ++++++++++ 4 files changed, 191 insertions(+), 12 deletions(-)
drivers/fpga/dfl-fme-main.c +54 -0
@@ -16,6 +16,7 @@#include <linux/kernel.h> #include <linux/module.h> +#include <linux/uaccess.h> #include <linux/fpga-dfl.h> #include "dfl.h"
@@ -105,9 +106,62 @@ static void fme_hdr_uinit(struct platform_device *pdev,sysfs_remove_files (&pdev->dev.kobj, fme_hdr_attrs); } +static long fme_hdr_ioctl_release_port (struct dfl_feature_platform_data *pdata, +void __user *arg) +{ +struct dfl_fpga_cdev *cdev = pdata->dfl_cdev; +struct dfl_fpga_fme_port_release release; +unsigned long minsz; + + minsz = offsetofend(struct dfl_fpga_fme_port_release, port_id); + +if (copy_from_user (&release, arg, minsz)) +return -EFAULT ; + +if (release.argsz < minsz || release.flags) +return -EINVAL ; + +return dfl_fpga_cdev_config_port (cdev, release.port_id, true); +} + +static long fme_hdr_ioctl_assign_port (struct dfl_feature_platform_data *pdata, +void __user *arg) +{ +struct dfl_fpga_cdev *cdev = pdata->dfl_cdev; +struct dfl_fpga_fme_port_assign assign; +unsigned long minsz; + + minsz = offsetofend(struct dfl_fpga_fme_port_assign, port_id); + +if (copy_from_user (&assign, arg, minsz)) +return -EFAULT ; + +if (assign.argsz < minsz || assign.flags) +return -EINVAL ; + +return dfl_fpga_cdev_config_port (cdev, assign.port_id, false); +} + +static long fme_hdr_ioctl (struct platform_device *pdev, +struct dfl_feature *feature, +unsigned int cmd,unsigned long arg) +{ +struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev); + +switch (cmd) { +case DFL_FPGA_FME_PORT_RELEASE : +return fme_hdr_ioctl_release_port (pdata, (void __user *)arg); +case DFL_FPGA_FME_PORT_ASSIGN : +return fme_hdr_ioctl_assign_port (pdata, (void __user *)arg); + } + +return -ENODEV ; +} +static const struct dfl_feature_ops fme_hdr_ops = { .init = fme_hdr_init, .uinit = fme_hdr_uinit, + .ioctl = fme_hdr_ioctl, };static struct dfl_feature_driver fme_feature_drvs[] = {
drivers/fpga/dfl.c +95 -12
148 lines not on this page · raw message
drivers/fpga/dfl.h +10 -0
@@ -183,6 +183,8 @@ struct dfl_feature {#define DEV_STATUS_IN_USE 0 +#define FEATURE_DEV_ID_UNUSED (-1) +/** * struct dfl_feature_platform_data - platform data for feature devices *
@@ -191,6 +193,7 @@ struct dfl_feature {* @cdev: cdev of feature dev. * @dev: ptr to platform device linked with this platform data. * @dfl_cdev: ptr to container device. +* @id: id used for this feature device. * @disable_count: count for port disable. * @num: number for sub features. * @dev_status: dev status (e.g. DEV_STATUS_IN_USE).
@@ -203,6 +206,7 @@ struct dfl_feature_platform_data {struct cdev cdev;struct platform_device *dev;struct dfl_fpga_cdev *dfl_cdev; +int id;unsigned int disable_count;unsigned long dev_status;void *private;
@@ -378,6 +382,7 @@ void dfl_fpga_enum_info_free(struct dfl_fpga_enum_info *info);* @fme_dev: FME feature device under this container device. * @lock: mutex lock to protect the port device list. * @port_dev_list: list of all port feature devices under this container device. +* @released_port_num: released port number under this container device. */ struct dfl_fpga_cdev {struct device *parent;
@@ -385,6 +390,7 @@ struct dfl_fpga_cdev {struct device *fme_dev;struct mutex lock;struct list_head port_dev_list; +int released_port_num; }; struct dfl_fpga_cdev *
@@ -412,4 +418,8 @@ dfl_fpga_cdev_find_port(struct dfl_fpga_cdev *cdev, void *data, return pdev; } + +int dfl_fpga_cdev_config_port (struct dfl_fpga_cdev *cdev, +u32 port_id,bool release); +#endif /* __FPGA_DFL_H */
include/uapi/linux/fpga-dfl.h +32 -0
@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release) +* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */ +__u32 flags;/* Zero for now */ +__u32 port_id; +}; + +#define DFL_FPGA_FME_PORT_RELEASE _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 1) + +/** +* DFL_FPGA_FME_PORT_ASSIGN - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 2, +* struct dfl_fpga_fme_port_assign) +* +* Driver assigns the port back per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_assign { +/* Input */ +__u32 argsz;/* Structure length */ +__u32 flags;/* Zero for now */ +__u32 port_id; +}; + +#define DFL_FPGA_FME_PORT_ASSIGN _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 2) +#endif /* _UAPI_LINUX_FPGA_DFL_H */
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 7/15 fpga: dfl: pci: enable SRIOV support.
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch enables the standard sriov support. It allows user to enable SRIOV (and VFs), then user could pass through accelerators (VFs) into virtual machine or use VFs directly in host.
--- drivers/fpga/dfl-pci.c | 40 ++++++++++++++++++++++++++++++++++++++++ drivers/fpga/dfl.c | 41 +++++++++++++++++++++++++++++++++++++++++ drivers/fpga/dfl.h | 1 + 3 files changed, 82 insertions(+)
drivers/fpga/dfl-pci.c +40 -0
@@ -223,8 +223,46 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)return ret; } +static int cci_pci_sriov_configure (struct pci_dev *pcidev,int num_vfs) +{ +struct cci_drvdata *drvdata =pci_get_drvdata (pcidev); +struct dfl_fpga_cdev *cdev = drvdata->cdev; +int ret =0 ; + +mutex_lock (&cdev->lock); + +if (!num_vfs) { +/* +* disable SRIOV and then put released ports back to default +* PF access mode. +*/ +pci_disable_sriov (pcidev); + +__dfl_fpga_cdev_config_port_vf (cdev, false); + + }else if (cdev->released_port_num == num_vfs) { +/* +* only enable SRIOV if cdev has matched released ports, put +* released ports into VF access mode firstly. +*/ +__dfl_fpga_cdev_config_port_vf (cdev, true); + + ret =pci_enable_sriov (pcidev, num_vfs); +if (ret) +__dfl_fpga_cdev_config_port_vf (cdev, false); + }else { + ret = -EINVAL ; + } + +mutex_unlock (&cdev->lock); +return ret; +} +static void cci_pci_remove (struct pci_dev *pcidev) { +if (dev_is_pf (&pcidev->dev)) +cci_pci_sriov_configure (pcidev,0 ); +cci_remove_feature_devs (pcidev);pci_disable_pcie_error_reporting (pcidev); }
@@ -234,6 +272,7 @@ static struct pci_driver cci_pci_driver = { .id_table = cci_pcie_id_tbl, .probe = cci_pci_probe, .remove = cci_pci_remove, + .sriov_configure = cci_pci_sriov_configure, };module_pci_driver (cci_pci_driver);
@@ -241,3 +280,4 @@ module_pci_driver(cci_pci_driver);MODULE_DESCRIPTION ("FPGA DFL PCIe Device Driver" );MODULE_AUTHOR ("Intel Corporation" );MODULE_LICENSE ("GPL v2" ); +MODULE_VERSION (DRV_VERSION );
drivers/fpga/dfl.c +41 -0
47 lines not on this page · raw message
drivers/fpga/dfl.h +1 -0
@@ -421,5 +421,6 @@ dfl_fpga_cdev_find_port(struct dfl_fpga_cdev *cdev, void *data,int dfl_fpga_cdev_config_port (struct dfl_fpga_cdev *cdev,u32 port_id,bool release); +void __dfl_fpga_cdev_config_port_vf (struct dfl_fpga_cdev *cdev,bool is_vf);#endif /* __FPGA_DFL_H */
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 8/15 fpga: dfl: afu: add AFU state related sysfs interfaces
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch introduces more sysfs interfaces for Accelerated Function Unit (AFU). These interfaces allow users to read current AFU Power State (APx), read / clear AFU Power (APx) events which are sticky to identify transient APx state, and manage AFU's LTR (latency tolerance reporting).
--- .../ABI/testing/sysfs-platform-dfl-port | 30 ++++ drivers/fpga/dfl-afu-main.c | 140 ++++++++++++++++++ drivers/fpga/dfl.h | 11 ++ 3 files changed, 181 insertions(+)
Documentation/ABI/testing/sysfs-platform-dfl-port +30 -0
@@ -14,3 +14,33 @@ Description: Read-only. User can program different PR bitstreams to FPGA Accelerator Function Unit (AFU) for different functions. It returns uuid which could be used to identify which PR bitstream is programmed in this AFU. + +What: /sys/bus/platform/devices/dfl-port.0/power_state +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. It reports the APx (AFU Power) state, different APx + means different throttling level. When reading this file, it + returns "0" - Normal / "1" - AP1 / "2" - AP2 / "6" - AP6. + +What: /sys/bus/platform/devices/dfl-port.0/ap1_event +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-write. Read or set 1 to clear AP1 (AFU Power State 1) + event. It's used to indicate transient AP1 state. + +What: /sys/bus/platform/devices/dfl-port.0/ap2_event +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-write. Read or set 1 to clear AP2 (AFU Power State 2) + event. It's used to indicate transient AP2 state. + +What: /sys/bus/platform/devices/dfl-port.0/ltr +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-write. Read and set AFU latency tolerance reporting value. + Set ltr to 1 if the AFU can tolerate latency >= 40us or set it + to 0 if it is latency sensitive.
drivers/fpga/dfl-afu-main.c +140 -0
157 lines not on this page · raw message
drivers/fpga/dfl.h +11 -0
@@ -119,6 +119,7 @@#define PORT_HDR_NEXT_AFU NEXT_AFU#define PORT_HDR_CAP 0x30#define PORT_HDR_CTRL 0x38 +#define PORT_HDR_STS 0x40/* Port Capability Register Bitfield */ #define PORT_CAP_PORT_NUM GENMASK_ULL(1, 0)/* ID of this port */
@@ -130,6 +131,16 @@/* Latency tolerance reporting. '1' >= 40us, '0' < 40us.*/ #define PORT_CTRL_LATENCY BIT_ULL(2)#define PORT_CTRL_SFTRST_ACK BIT_ULL(4)/* HW ack for reset */ + +/* Port Status Register Bitfield */ +#define PORT_STS_AP2_EVT BIT_ULL(13)/* AP2 event detected */ +#define PORT_STS_AP1_EVT BIT_ULL(12)/* AP1 event detected */ +#define PORT_STS_PWR_STATE GENMASK_ULL(11, 8)/* AFU power states */ +#define PORT_STS_PWR_STATE_NORM 0 +#define PORT_STS_PWR_STATE_AP1 1/* 50% throttling */ +#define PORT_STS_PWR_STATE_AP2 2/* 90% throttling */ +#define PORT_STS_PWR_STATE_AP6 6/* 100% throttling */ +/** * struct dfl_fpga_port_ops - port ops *
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 9/15 fpga: dfl: afu: add userclock sysfs interfaces.
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch introduces userclock sysfs interfaces for AFU, user could use these interfaces for clock setting to AFU.
Please note that, this is only working for port header feature with revision 0, for later revisions, userclock setting is moved to a separated private feature, so one revision sysfs interface is exposed to userspace application for this purpose too.
--- .../ABI/testing/sysfs-platform-dfl-port | 35 ++++++ drivers/fpga/dfl-afu-main.c | 113 +++++++++++++++++- drivers/fpga/dfl.h | 4 + 3 files changed, 151 insertions(+), 1 deletion(-)
Documentation/ABI/testing/sysfs-platform-dfl-port +35 -0
@@ -44,3 +44,38 @@ Contact: Wu Hao <[email protected]> Description: Read-write. Read and set AFU latency tolerance reporting value. Set ltr to 1 if the AFU can tolerate latency >= 40us or set it to 0 if it is latency sensitive. + +What: /sys/bus/platform/devices/dfl-port.0/revision +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the revision of port header + feature. + +What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcmd +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Write-only. User writes command to this interface to set + userclock to AFU. + +What: /sys/bus/platform/devices/dfl-port.0/userclk_freqsts +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the status of issued command + to userclck_freqcmd. + +What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrcmd +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Write-only. User writes command to this interface to set + userclock counter. + +What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrsts +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the status of issued command + to userclck_freqcntrcmd.
drivers/fpga/dfl-afu-main.c +112 -1
144 lines not on this page · raw message
drivers/fpga/dfl.h +4 -0
@@ -120,6 +120,10 @@#define PORT_HDR_CAP 0x30#define PORT_HDR_CTRL 0x38#define PORT_HDR_STS 0x40 +#define PORT_HDR_USRCLK_CMD0 0x50 +#define PORT_HDR_USRCLK_CMD1 0x58 +#define PORT_HDR_USRCLK_STS0 0x60 +#define PORT_HDR_USRCLK_STS1 0x68/* Port Capability Register Bitfield */ #define PORT_CAP_PORT_NUM GENMASK_ULL(1, 0)/* ID of this port */
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 10/15 fpga: dfl: add id_table for dfl private feature driver
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch adds id_table for each dfl private feature driver, it allows to reuse same private feature driver to match and support multiple dfl private features.
--- drivers/fpga/dfl-afu-main.c | 14 ++++++++++++-- drivers/fpga/dfl-fme-main.c | 11 ++++++++--- drivers/fpga/dfl-fme-pr.c | 7 ++++++- drivers/fpga/dfl-fme.h | 3 ++- drivers/fpga/dfl.c | 21 +++++++++++++++++++-- drivers/fpga/dfl.h | 21 +++++++++++++++------ 6 files changed, 62 insertions(+), 15 deletions(-)
drivers/fpga/dfl-afu-main.c +12 -2
@@ -435,6 +435,11 @@ port_hdr_ioctl(struct platform_device *pdev, struct dfl_feature *feature,return ret; } +static const struct dfl_feature_id port_hdr_id_table[] = { + {.id =PORT_FEATURE_ID_HEADER ,}, + {0 ,} +}; +static const struct dfl_feature_ops port_hdr_ops = { .init = port_hdr_init, .uinit = port_hdr_uinit,
@@ -495,6 +500,11 @@ static void port_afu_uinit(struct platform_device *pdev,sysfs_remove_files (&pdev->dev.kobj, port_afu_attrs); } +static const struct dfl_feature_id port_afu_id_table[] = { + {.id =PORT_FEATURE_ID_AFU ,}, + {0 ,} +}; +static const struct dfl_feature_ops port_afu_ops = { .init = port_afu_init, .uinit = port_afu_uinit,
@@ -502,11 +512,11 @@ static const struct dfl_feature_ops port_afu_ops = {static struct dfl_feature_driver port_feature_drvs[] = { { - .id =PORT_FEATURE_ID_HEADER , + .id_table = port_hdr_id_table, .ops = &port_hdr_ops, }, { - .id =PORT_FEATURE_ID_AFU , + .id_table = port_afu_id_table, .ops = &port_afu_ops, }, {
drivers/fpga/dfl-fme-main.c +8 -3
@@ -158,6 +158,11 @@ static long fme_hdr_ioctl(struct platform_device *pdev,return -ENODEV ; } +static const struct dfl_feature_id fme_hdr_id_table[] = { + {.id =FME_FEATURE_ID_HEADER ,}, + {0 ,} +}; +static const struct dfl_feature_ops fme_hdr_ops = { .init = fme_hdr_init, .uinit = fme_hdr_uinit,
@@ -166,12 +171,12 @@ static const struct dfl_feature_ops fme_hdr_ops = {static struct dfl_feature_driver fme_feature_drvs[] = { { - .id =FME_FEATURE_ID_HEADER , + .id_table = fme_hdr_id_table, .ops = &fme_hdr_ops, }, { - .id =FME_FEATURE_ID_PR_MGMT , - .ops = &pr_mgmt_ops, + .id_table = fme_pr_mgmt_id_table, + .ops = &fme_pr_mgmt_ops, }, { .ops =NULL ,
drivers/fpga/dfl-fme-pr.c +6 -1
@@ -483,7 +483,12 @@ static long fme_pr_ioctl(struct platform_device *pdev,return ret; } -const struct dfl_feature_ops pr_mgmt_ops = { +const struct dfl_feature_id fme_pr_mgmt_id_table[] = { + {.id =FME_FEATURE_ID_PR_MGMT ,}, + {0 } +}; + +const struct dfl_feature_ops fme_pr_mgmt_ops = { .init = pr_mgmt_init, .uinit = pr_mgmt_uinit, .ioctl = fme_pr_ioctl,
drivers/fpga/dfl-fme.h +2 -1
@@ -35,6 +35,7 @@ struct dfl_fme { struct dfl_feature_platform_data *pdata; }; -extern const struct dfl_feature_ops pr_mgmt_ops; +extern const struct dfl_feature_ops fme_pr_mgmt_ops; +extern const struct dfl_feature_id fme_pr_mgmt_id_table[];#endif /* __DFL_FME_H */
drivers/fpga/dfl.c +19 -2
@@ -14,6 +14,8 @@#include "dfl.h" +#define DRV_VERSION "0.8" +static DEFINE_MUTEX (dfl_id_mutex);/*
@@ -281,6 +283,21 @@ static int dfl_feature_instance_init(struct platform_device *pdev,return ret; } +static bool dfl_feature_drv_match (struct dfl_feature *feature, +struct dfl_feature_driver *driver) +{ +const struct dfl_feature_id *ids = driver->id_table; + +if (ids) { +while (ids->id) { +if (ids->id == feature->id) +return true; + ids++; + } + } +return false; +} +/** * dfl_fpga_dev_feature_init - init for sub features of dfl feature device * @pdev: feature device.
@@ -301,8 +318,7 @@ int dfl_fpga_dev_feature_init(struct platform_device *pdev,while (drv->ops) {dfl_fpga_dev_for_each_feature (pdata, feature) { -/* match feature and drv using id */ -if (feature->id == drv->id) { +if (dfl_feature_drv_match (feature, drv)) { ret =dfl_feature_instance_init (pdev, pdata, feature, drv);if (ret)
@@ -1178,3 +1194,4 @@ module_exit(dfl_fpga_exit);MODULE_DESCRIPTION ("FPGA Device Feature List (DFL) Support" );MODULE_AUTHOR ("Intel Corporation" );MODULE_LICENSE ("GPL v2" ); +MODULE_VERSION (DRV_VERSION );
drivers/fpga/dfl.h +15 -6
@@ -30,8 +30,8 @@/* plus one for fme device */ #define MAX_DFL_FEATURE_DEV_NUM (MAX_DFL_FPGA_PORT_NUM + 1) -/* Reserved 0x0 for Header Group Register and 0xff for AFU */ -#define FEATURE_ID_FIU_HEADER 0x0 +/* Reserved 0xfe for Header Group Register and 0xff for AFU */ +#define FEATURE_ID_FIU_HEADER 0xfe#define FEATURE_ID_AFU 0xff#define FME_FEATURE_ID_HEADER FEATURE_ID_FIU_HEADER
@@ -169,13 +169,22 @@ void dfl_fpga_port_ops_put(struct dfl_fpga_port_ops *ops);int dfl_fpga_check_port_id (struct platform_device *pdev,void *pport_id);/** -* struct dfl_feature_driver - sub feature's driver +* struct dfl_feature_id - dfl private feature id * -* @id: sub feature id. -* @ops: ops of this sub feature. +* @id: unique dfl private feature id. */ -struct dfl_feature_driver { +struct dfl_feature_id {u64 id; +}; + +/** +* struct dfl_feature_driver - dfl private feature driver +* +* @id_table: id_table for dfl private features supported by this driver. +* @ops: ops of this dfl private feature driver. +*/ +struct dfl_feature_driver { +const struct dfl_feature_id *id_table;const struct dfl_feature_ops *ops; };
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 11/15 fpga: dfl: afu: export __port_enable/disable function.
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
As these two functions are used by other private features. e.g. in error reporting private feature, it requires to check port status and reset port for error clearing.
--- drivers/fpga/dfl-afu-main.c | 25 ++++++++++++++----------- drivers/fpga/dfl-afu.h | 3 +++ 2 files changed, 17 insertions(+), 11 deletions(-)
drivers/fpga/dfl-afu-main.c +14 -11
@@ -24,14 +24,16 @@#define DRV_VERSION "0.8"/** -* port_enable - enable a port +* __port_enable - enable a port * @pdev: port platform device. * * Enable Port by clear the port soft reset bit, which is set by default. * The AFU is unable to respond to any MMIO access while in reset. -* port_enable function should only be used after port_disable function. +* __port_enable function should only be used after __port_disable function. +* +* The caller needs to hold lock for protection. */ -static void port_enable (struct platform_device *pdev) +void __port_enable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;
@@ -54,13 +56,14 @@ static void port_enable(struct platform_device *pdev)#define RST_POLL_TIMEOUT 1000/* us */ /** -* port_disable - disable a port +* __port_disable - disable a port * @pdev: port platform device. * -* Disable Port by setting the port soft reset bit, it puts the port into -* reset. +* Disable Port by setting the port soft reset bit, it puts the port into reset. +* +* The caller needs to hold lock for protection. */ -static int port_disable (struct platform_device *pdev) +int __port_disable (struct platform_device *pdev) {struct dfl_feature_platform_data *pdata =dev_get_platdata (&pdev->dev);void __iomem *base;
@@ -106,9 +109,9 @@ static int __port_reset(struct platform_device *pdev) {int ret; - ret =port_disable (pdev); + ret =__port_disable (pdev);if (!ret) -port_enable (pdev); +__port_enable (pdev);return ret; }
@@ -805,9 +808,9 @@ static int port_enable_set(struct platform_device *pdev, bool enable)mutex_lock (&pdata->lock);if (enable) -port_enable (pdev); +__port_enable (pdev);else - ret =port_disable (pdev); + ret =__port_disable (pdev);mutex_unlock (&pdata->lock);return ret;
drivers/fpga/dfl-afu.h +3 -0
@@ -79,6 +79,9 @@ struct dfl_afu {struct dfl_feature_platform_data *pdata; }; +void __port_enable (struct platform_device *pdev); +int __port_disable (struct platform_device *pdev); +void afu_mmio_region_init (struct dfl_feature_platform_data *pdata);int afu_mmio_region_add (struct dfl_feature_platform_data *pdata,u32 region_index,u64 region_size,u64 phys,u32 flags);
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 12/15 fpga: dfl: afu: add error reporting support.
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
Error reporting is one important private feature, it reports error detected on port and accelerated function unit (AFU). It introduces several sysfs interfaces to allow userspace to check and clear errors detected by hardware.
--- .../ABI/testing/sysfs-platform-dfl-port | 39 +++ drivers/fpga/Makefile | 1 + drivers/fpga/dfl-afu-error.c | 225 ++++++++++++++++++ drivers/fpga/dfl-afu-main.c | 4 + drivers/fpga/dfl-afu.h | 4 + 5 files changed, 273 insertions(+) create mode 100644 drivers/fpga/dfl-afu-error.c
Documentation/ABI/testing/sysfs-platform-dfl-port +39 -0
@@ -79,3 +79,42 @@ KernelVersion: 5.3 Contact: Wu Hao <[email protected]> Description: Read-only. Read this file to get the status of issued command to userclck_freqcntrcmd. + +What: /sys/bus/platform/devices/dfl-port.0/errors/revision +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the revision of this error + reporting private feature. + +What: /sys/bus/platform/devices/dfl-port.0/errors/errors +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get errors detected on port and + Accelerated Function Unit (AFU). + +What: /sys/bus/platform/devices/dfl-port.0/errors/first_error +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the first error detected by + hardware. + +What: /sys/bus/platform/devices/dfl-port.0/errors/first_malformed_req +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the first malformed request + captured by hardware. + +What: /sys/bus/platform/devices/dfl-port.0/errors/clear +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Write-only. Write error code to this file to clear errors. + Write fails with -EINVAL if input parsing fails or input error + code doesn't match. + Write fails with -EBUSY or -ETIMEDOUT if error can't be cleared + as hardware is in low power state (-EBUSY) or not responding + (-ETIMEDOUT).
drivers/fpga/Makefile +1 -0
@@ -41,6 +41,7 @@ obj-$(CONFIG_FPGA_DFL_AFU) += dfl-afu.o dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o dfl-afu-dma-region.o +dfl-afu-objs += dfl-afu-error.o # Drivers for FPGAs which implement DFL obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
drivers/fpga/dfl-afu-error.c new file +225 -0
225 lines not on this page · raw message
drivers/fpga/dfl-afu-main.c +4 -0
@@ -522,6 +522,10 @@ static struct dfl_feature_driver port_feature_drvs[] = { .id_table = port_afu_id_table, .ops = &port_afu_ops, }, + { + .id_table = port_err_id_table, + .ops = &port_err_ops, + }, { .ops = NULL, }
drivers/fpga/dfl-afu.h +4 -0
@@ -100,4 +100,8 @@ int afu_dma_unmap_region(struct dfl_feature_platform_data *pdata, u64 iova);struct dfl_afu_dma_region *afu_dma_region_find (struct dfl_feature_platform_data *pdata,u64 iova,u64 size); + +extern const struct dfl_feature_ops port_err_ops; +extern const struct dfl_feature_id port_err_id_table[]; +#endif /* __DFL_AFU_H */
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 13/15 fpga: dfl: afu: add STP (SignalTap) support
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
STP (SignalTap) is one of the private features under the port for debugging. This patch adds private feature driver support for it to allow userspace applications to mmap related mmio region and provide STP service.
--- drivers/fpga/dfl-afu-main.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/fpga/dfl-afu-main.c +34 -0
@@ -513,6 +513,36 @@ static const struct dfl_feature_ops port_afu_ops = { .uinit = port_afu_uinit, }; +static int port_stp_init (struct platform_device *pdev, +struct dfl_feature *feature) +{ +struct resource *res = &pdev->resource[feature->resource_index]; + +dev_dbg (&pdev->dev,"PORT STP Init.\n" ); + +return afu_mmio_region_add (dev_get_platdata (&pdev->dev), +DFL_PORT_REGION_INDEX_STP , +resource_size (res), res->start, +DFL_PORT_REGION_MMAP |DFL_PORT_REGION_READ | +DFL_PORT_REGION_WRITE ); +} + +static void port_stp_uinit (struct platform_device *pdev, +struct dfl_feature *feature) +{ +dev_dbg (&pdev->dev,"PORT STP UInit.\n" ); +} + +static const struct dfl_feature_id port_stp_id_table[] = { + {.id =PORT_FEATURE_ID_STP ,}, + {0 ,} +}; + +static const struct dfl_feature_ops port_stp_ops = { + .init = port_stp_init, + .uinit = port_stp_uinit, +}; +static struct dfl_feature_driver port_feature_drvs[] = { { .id_table = port_hdr_id_table,
@@ -526,6 +556,10 @@ static struct dfl_feature_driver port_feature_drvs[] = { .id_table = port_err_id_table, .ops = &port_err_ops, }, + { + .id_table = port_stp_id_table, + .ops = &port_stp_ops, + }, { .ops = NULL, }
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 14/15 fpga: dfl: fme: add capability sysfs interfaces
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch adds 3 read-only sysfs interfaces for FPGA Management Engine (FME) block for capabilities including cache_size, fabric_version and socket_id.
--- .../ABI/testing/sysfs-platform-dfl-fme | 23 +++++++++ drivers/fpga/dfl-fme-main.c | 48 +++++++++++++++++++ 2 files changed, 71 insertions(+)
Documentation/ABI/testing/sysfs-platform-dfl-fme +23 -0
@@ -21,3 +21,26 @@ Contact: Wu Hao <[email protected]> Description: Read-only. It returns Bitstream (static FPGA region) meta data, which includes the synthesis date, seed and other information of this static FPGA region. + +What: /sys/bus/platform/devices/dfl-fme.0/cache_size +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. It returns cache size of this FPGA device. + +What: /sys/bus/platform/devices/dfl-fme.0/fabric_version +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. It returns fabric version of this FPGA device. + Userspace applications need this information to select + best data channels per different fabric design. + +What: /sys/bus/platform/devices/dfl-fme.0/socket_id +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. It returns socket_id to indicate which socket + this FPGA belongs to, only valid for integrated solution. + User only needs this information, in case standard numa node + can't provide correct information.
drivers/fpga/dfl-fme-main.c +48 -0
58 lines not on this page · raw message
Signature · 1 line
2.22.0
Moritz Fischer
maintainerPATCH 15/15 fpga: dfl: fme: add global error reporting support
In reply to Moritz FischerFrom: Wu Hao <[email protected]>
This patch adds support for global error reporting for FPGA Management Engine (FME), it introduces sysfs interfaces to report different error detected by the hardware, and allow user to clear errors or inject error for testing purpose.
--- .../ABI/testing/sysfs-platform-dfl-fme | 75 ++++ drivers/fpga/Makefile | 2 +- drivers/fpga/dfl-fme-error.c | 385 ++++++++++++++++++ drivers/fpga/dfl-fme-main.c | 4 + drivers/fpga/dfl-fme.h | 2 + drivers/fpga/dfl.h | 2 + 6 files changed, 469 insertions(+), 1 deletion(-) create mode 100644 drivers/fpga/dfl-fme-error.c
Documentation/ABI/testing/sysfs-platform-dfl-fme +75 -0
@@ -44,3 +44,78 @@ Description: Read-only. It returns socket_id to indicate which socket this FPGA belongs to, only valid for integrated solution. User only needs this information, in case standard numa node can't provide correct information. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/revision +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the revision of this global + error reporting private feature. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/pcie0_errors +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-Write. Read this file for errors detected on pcie0 link. + Write this file to clear errors logged in pcie0_errors. Write + fails with -EINVAL if input parsing fails or input error code + doesn't match. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/pcie1_errors +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-Write. Read this file for errors detected on pcie1 link. + Write this file to clear errors logged in pcie1_errors. Write + fails with -EINVAL if input parsing fails or input error code + doesn't match. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/nonfatal_errors +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. It returns non-fatal errors detected. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/catfatal_errors +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. It returns catastrophic and fatal errors detected. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/inject_error +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-Write. Read this file to check errors injected. Write this + file to inject errors for testing purpose. Write fails with + -EINVAL if input parsing fails or input inject error code isn't + supported. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/errors +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get errors detected by hardware. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/first_error +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the first error detected by + hardware. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/next_error +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Read-only. Read this file to get the second error detected by + hardware. + +What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/clear +Date: June 2019 +KernelVersion: 5.3 +Contact: Wu Hao <[email protected]> +Description: Write-only. Write error code to this file to clear all errors + logged in errors, first_error and next_error. Write fails with + -EINVAL if input parsing fails or input error code doesn't + match.
drivers/fpga/Makefile +1 -1
@@ -39,7 +39,7 @@ obj-$(CONFIG_FPGA_DFL_FME_BRIDGE) += dfl-fme-br.o obj-$(CONFIG_FPGA_DFL_FME_REGION) += dfl-fme-region.o obj-$(CONFIG_FPGA_DFL_AFU) += dfl-afu.o -dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o +dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o dfl-fme-error.o dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o dfl-afu-dma-region.o dfl-afu-objs += dfl-afu-error.o
drivers/fpga/dfl-fme-error.c new file +385 -0
385 lines not on this page · raw message
drivers/fpga/dfl-fme-main.c +4 -0
@@ -226,6 +226,10 @@ static struct dfl_feature_driver fme_feature_drvs[] = { .id_table = fme_pr_mgmt_id_table, .ops = &fme_pr_mgmt_ops, }, + { + .id_table = fme_global_err_id_table, + .ops = &fme_global_err_ops, + }, { .ops = NULL, },
drivers/fpga/dfl-fme.h +2 -0
@@ -37,5 +37,7 @@ struct dfl_fme {extern const struct dfl_feature_ops fme_pr_mgmt_ops;extern const struct dfl_feature_id fme_pr_mgmt_id_table[]; +extern const struct dfl_feature_ops fme_global_err_ops; +extern const struct dfl_feature_id fme_global_err_id_table[];#endif /* __DFL_FME_H */
drivers/fpga/dfl.h +2 -0
@@ -197,12 +197,14 @@ struct dfl_feature_driver {* feature dev (platform device)'s reources. * @ioaddr: mapped mmio resource address. * @ops: ops of this sub feature. +* @priv: priv data of this feature. */ struct dfl_feature {u64 id;int resource_index;void __iomem *ioaddr;const struct dfl_feature_ops *ops; +void *priv; };#define DEV_STATUS_IN_USE 0
Signature · 1 line
2.22.0
Re: [PATCH 04/15] fpga: dfl: fme: support 512bit data width PR
In reply to Moritz FischerQuoted 44 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In early partial reconfiguration private feature, it only supports 32bit data width when writing data to hardware for PR. 512bit data width PR support is an important optimization for some specific solutions (e.g. XEON with FPGA integrated), it allows driver to use AVX512 instruction to improve the performance of partial reconfiguration. e.g. programming one 100MB bitstream image via this 512bit data width PR hardware only takes ~300ms, but 32bit revision requires ~3s per test result.
Please note now this optimization is only done on revision 2 of this PR private feature which is only used in integrated solution that AVX512 is always supported. This revision 2 hardware doesn't support 32bit PR.
Signed-off-by Ananda Ravuri <[email protected]>Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- drivers/fpga/dfl-fme-main.c | 3 + drivers/fpga/dfl-fme-mgr.c | 113 +++++++++++++++++++++++++++++++----- drivers/fpga/dfl-fme-pr.c | 43 +++++++++----- drivers/fpga/dfl-fme.h | 2 + drivers/fpga/dfl.h | 5 ++ 5 files changed, 135 insertions(+), 31 deletions(-)drivers/fpga/dfl-fme-main.c +2
-0@@ -21,6 +21,8 @@#include "dfl.h" #include "dfl-fme.h" +#define DRV_VERSION "0.8" +static ssize_t ports_num_show (struct device *dev,struct device_attribute *attr,char *buf) {
drivers/fpga/dfl-fme-main.c MODULE_DESCRIPTION("FPGA Management Engine driver"); · line 282
Quoted 4 lines
drivers/fpga/dfl-fme-main.c +1
-0@@ -277,3 +279,4 @@ MODULE_DESCRIPTION("FPGA Management Engine driver");MODULE_AUTHOR ("Intel Corporation" );MODULE_LICENSE ("GPL v2" );MODULE_ALIAS ("platform:dfl-fme" ); +MODULE_VERSION (DRV_VERSION );
No, we ripped out these useless "driver version" things all over the place, please do not add them back in again. They mean nothing and confuse people to no end.
I'll not take this patch, sorry.
greg k-h
Re: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
In reply to Moritz FischerQuoted 6 lines · Moritz Fischer
From: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
Quoted 8 lines
[[email protected]: Fixed up to make it work with new reStructuredText docs]Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)
This doesn't apply to my tree, where is this file created?
thanks,
greg k-h
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
In reply to Moritz FischerQuoted 17 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASEioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke thisPORT_ASSIGNioctl to attach the port back to PF.Ioctl interfaces:
*DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.
Quoted 4 lines
*
DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.
Why are you not using the "generic" bind/unbind facility that userspace already has for this with binding drivers to devices? Why a special ioctl?
Quoted 9 lines
include/uapi/linux/fpga-dfl.h +3
-0@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release)
include/uapi/linux/fpga-dfl.h struct dfl_fpga_fme_port_pr { · line 191
Quoted 10 lines
include/uapi/linux/fpga-dfl.h +10
-0+* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */ +__u32 flags;/* Zero for now */ +__u32 port_id; +};
meta-comment, why do all of your structures for ioctls have argsz? You "know" the size of the structure already, it's part of the ioctl definition. You shouldn't need to also set it again, right? Otherwise ALL Linux ioctls would need something crazy like this.
thanks,
greg k-h
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
Quoted 18 lines · Greg KH
Quoted 17 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASEioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke thisPORT_ASSIGNioctl to attach the port back to PF.Ioctl interfaces:
*DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.
Quoted 8 lines
Quoted 4 lines
*
DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.Why are you not using the "generic" bind/unbind facility that userspace already has for this with binding drivers to devices? Why a special ioctl?
Hi Greg,
Actually we think it should be safer that making the device invisble than just unbinding its driver. Looks like user can try to rebind it at any time and we don't have any method to stop them.
Quoted 17 lines
Quoted 17 lines
include/uapi/linux/fpga-dfl.h +10
-0@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release) +* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */
Quoted 8 lines
include/uapi/linux/fpga-dfl.h struct dfl_fpga_fme_port_pr { · line 191
Quoted 3 lines
include/uapi/linux/fpga-dfl.h +3
-0+__u32 flags;/* Zero for now */ +__u32 port_id; +};meta-comment, why do all of your structures for ioctls have argsz? You "know" the size of the structure already, it's part of the ioctl definition. You shouldn't need to also set it again, right? Otherwise ALL Linux ioctls would need something crazy like this.
Actually we followed the same method as vfio. The major purpose should be extendibility, as we really need this to be sth long term maintainable. It really helps, if we add some new members for extentions/enhancement under the same ioctl. I don't think everybody needs this, but my consideration here is if newer generations of hardware/specs come with some extentions, I still hope we can resue these IOCTLs as much as we could, instead of creating more new ones.
Thanks
Hao
Quoted 4 lines
thanks,
greg k-h
Re: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
In reply to Greg KHQuoted 16 lines · Greg KH
Quoted 6 lines · Moritz Fischer
From: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
Quoted 8 lines
[[email protected]: Fixed up to make it work with new reStructuredText docs]Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)
Quoted 1 line
This doesn't apply to my tree, where is this file created?
Hi Greg,
From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst in linux-next. I think this patch is created on top of that by Moritz.
"Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst")"
Thanks
Hao
Quoted 4 lines
thanks,
greg k-h
Re: [PATCH 04/15] fpga: dfl: fme: support 512bit data width PR
In reply to Greg KHQuoted 54 lines · Greg KH
Quoted 44 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In early partial reconfiguration private feature, it only supports 32bit data width when writing data to hardware for PR. 512bit data width PR support is an important optimization for some specific solutions (e.g. XEON with FPGA integrated), it allows driver to use AVX512 instruction to improve the performance of partial reconfiguration. e.g. programming one 100MB bitstream image via this 512bit data width PR hardware only takes ~300ms, but 32bit revision requires ~3s per test result.
Please note now this optimization is only done on revision 2 of this PR private feature which is only used in integrated solution that AVX512 is always supported. This revision 2 hardware doesn't support 32bit PR.
Signed-off-by Ananda Ravuri <[email protected]>Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- drivers/fpga/dfl-fme-main.c | 3 + drivers/fpga/dfl-fme-mgr.c | 113 +++++++++++++++++++++++++++++++----- drivers/fpga/dfl-fme-pr.c | 43 +++++++++----- drivers/fpga/dfl-fme.h | 2 + drivers/fpga/dfl.h | 5 ++ 5 files changed, 135 insertions(+), 31 deletions(-)drivers/fpga/dfl-fme-main.c +2
-0@@ -21,6 +21,8 @@#include "dfl.h" #include "dfl-fme.h" +#define DRV_VERSION "0.8" +static ssize_t ports_num_show (struct device *dev,struct device_attribute *attr,char *buf) {drivers/fpga/dfl-fme-main.c MODULE_DESCRIPTION("FPGA Management Engine driver"); · line 282
Quoted 4 lines
drivers/fpga/dfl-fme-main.c +1
-0@@ -277,3 +279,4 @@ MODULE_DESCRIPTION("FPGA Management Engine driver");MODULE_AUTHOR ("Intel Corporation" );MODULE_LICENSE ("GPL v2" );MODULE_ALIAS ("platform:dfl-fme" ); +MODULE_VERSION (DRV_VERSION );No, we ripped out these useless "driver version" things all over the place, please do not add them back in again. They mean nothing and confuse people to no end.
Quoted 1 line
I'll not take this patch, sorry.
Let me remove them from these patches, and generate a new version quickly.
Thanks for the review and comments.
Hao
Quoted 2 lines
greg k-h
Re: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
In reply to Wu HaoQuoted 19 lines · Wu Hao
Quoted 16 lines · Greg KH
Quoted 6 lines · Moritz Fischer
From: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
Quoted 8 lines
[[email protected]: Fixed up to make it work with new reStructuredText docs]Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)Quoted 1 line
This doesn't apply to my tree, where is this file created?
Quoted 7 lines
Hi Greg,
Quoted 1 line
From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
in linux-next. I think this patch is created on top of that by Moritz.
"Note: I've seen that Mauro touched
Documentation/fpga/dfl.rstin linux-next commitc220a1fae6c5d("docs: fpga: convert docs to ReST and rename to *.rst")"
Ok, but I can't take this patch just because the file is converted in someone else's tree :(
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
In reply to Wu HaoQuoted 28 lines · Wu Hao
Quoted 18 lines · Greg KH
Quoted 17 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASEioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke thisPORT_ASSIGNioctl to attach the port back to PF.Ioctl interfaces:
*DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.Quoted 8 lines
Quoted 4 lines
*
DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.Why are you not using the "generic" bind/unbind facility that userspace already has for this with binding drivers to devices? Why a special ioctl?
Quoted 5 lines
Hi Greg,
Actually we think it should be safer that making the device invisble than just unbinding its driver. Looks like user can try to rebind it at any time and we don't have any method to stop them.
Why do you want to "stop" the user from doing something? They asked to do it, why prevent it? If they ask to do something foolish, well, they get to keep the pieces :)
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
include/uapi/linux/fpga-dfl.h +10
-0@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release) +* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */ Quoted 8 lines
include/uapi/linux/fpga-dfl.h struct dfl_fpga_fme_port_pr { · line 191
Quoted 3 lines
include/uapi/linux/fpga-dfl.h +3
-0+__u32 flags;/* Zero for now */ +__u32 port_id; +};meta-comment, why do all of your structures for ioctls have argsz? You "know" the size of the structure already, it's part of the ioctl definition. You shouldn't need to also set it again, right? Otherwise ALL Linux ioctls would need something crazy like this.
Quoted 1 line
Actually we followed the same method as vfio.
vfio is a protocol on "the wire", right? Not an ioctl.
Quoted 2 lines
The major purpose should be extendibility, as we really need this to be sth long term maintainable.
You can't change ioctl structure sizes at any time.
Quoted 2 lines
It really helps, if we add some new members for extentions/enhancement under the same ioctl.
You don't do that.
Quoted 4 lines
I don't think everybody needs this, but my consideration here is if newer generations of hardware/specs come with some extentions, I still hope we can resue these IOCTLs as much as we could, instead of creating more new ones.
You create new ones, like everyone else does, as you can not change old code. By trying to "version" structures like this, it's just going to be a nightmare.
thanks,
greg k-h
Re: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
In reply to Greg KHQuoted 22 lines · Greg KH
Quoted 21 lines · Wu Hao
Quoted 16 lines · Greg KH
Quoted 6 lines · Moritz Fischer
From: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
Quoted 8 lines
[[email protected]: Fixed up to make it work with new reStructuredText docs]Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)Quoted 1 line
This doesn't apply to my tree, where is this file created?
Hi Greg,
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
in linux-next. I think this patch is created on top of that by Moritz.
"Note: I've seen that Mauro touched
Documentation/fpga/dfl.rstin linux-next commitc220a1fae6c5d("docs: fpga: convert docs to ReST and rename to *.rst")"Ok, but I can't take this patch just because the file is converted in someone else's tree :(
Oh.. if that patch is merged from fpga tree, then we wont have this problem. :(
So do you have any suggestions on what should i do now? wait that patch goes to offical release, and then resubmit this patch?
Hao
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
In reply to Greg KHQuoted 31 lines · Greg KH
Quoted 30 lines · Wu Hao
Quoted 18 lines · Greg KH
Quoted 17 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASEioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke thisPORT_ASSIGNioctl to attach the port back to PF.Ioctl interfaces:
*DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.Quoted 8 lines
Quoted 4 lines
*
DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.Why are you not using the "generic" bind/unbind facility that userspace already has for this with binding drivers to devices? Why a special ioctl?
Hi Greg,
Quoted 7 lines
Quoted 3 lines
Actually we think it should be safer that making the device invisble than just unbinding its driver. Looks like user can try to rebind it at any time and we don't have any method to stop them.
Why do you want to "stop" the user from doing something? They asked to do it, why prevent it? If they ask to do something foolish, well, they get to keep the pieces :)
Actually this is for SRIOV support, as we are moving FPGA accelerator from PF to VF, so we don't want users to see the FPGA accelerator from PF any more. We can't allow user to touch same FPGA accelerator from both PF and VF side (it leads to hardware erros).
Quoted 40 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
include/uapi/linux/fpga-dfl.h +10
-0@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release) +* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */ Quoted 8 lines
include/uapi/linux/fpga-dfl.h struct dfl_fpga_fme_port_pr { · line 191
Quoted 3 lines
include/uapi/linux/fpga-dfl.h +3
-0+__u32 flags;/* Zero for now */ +__u32 port_id; +};meta-comment, why do all of your structures for ioctls have argsz? You "know" the size of the structure already, it's part of the ioctl definition. You shouldn't need to also set it again, right? Otherwise ALL Linux ioctls would need something crazy like this.
Quoted 1 line
Actually we followed the same method as vfio.
vfio is a protocol on "the wire", right? Not an ioctl.
Quoted 2 lines
The major purpose should be extendibility, as we really need this to be sth long term maintainable.
You can't change ioctl structure sizes at any time.
Quoted 2 lines
It really helps, if we add some new members for extentions/enhancement under the same ioctl.
You don't do that.
Quoted 8 lines
Quoted 4 lines
I don't think everybody needs this, but my consideration here is if newer generations of hardware/specs come with some extentions, I still hope we can resue these IOCTLs as much as we could, instead of creating more new ones.
You create new ones, like everyone else does, as you can not change old code. By trying to "version" structures like this, it's just going to be a nightmare.
Actually i learned this from vfio code here, it's not trying to "version" structures, let me copy the comments from vfio header file. It should be more clear than above short description from me.
"include/uapi/linux/vfio.h"
/*
* The IOCTL interface is designed for extensibility by embedding the
* structure length (argsz) and flags into structures passed between
* kernel and userspace. We therefore use the _IO() macro for these
* defines to avoid implicitly embedding a size into the ioctl request.
* As structure fields are added, argsz will increase to match and flag
* bits will be defined to indicate additional fields with valid data.
* It's *always* the caller's responsibility to indicate the size of
* the structure passed by setting argsz appropriately.
*/
For example.
struct vfio_device_info {
__u32 argsz;
__u32 flags;
#define VFIO_DEVICE_FLAGS_RESET (1 << 0) /* Device supports reset */
#define VFIO_DEVICE_FLAGS_PCI (1 << 1) /* vfio-pci device */
#define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */
#define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
#define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */Hope things could be more clear now. :)
Thanks
Hao
};
Quoted 4 lines
thanks,
greg k-h
Re: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
In reply to Wu HaoQuoted 26 lines · Wu Hao
Quoted 25 lines · Greg KH
Quoted 24 lines · Wu Hao
Quoted 16 lines · Greg KH
Quoted 6 lines · Moritz Fischer
From: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
Quoted 8 lines
[[email protected]: Fixed up to make it work with new reStructuredText docs]Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)Quoted 1 line
This doesn't apply to my tree, where is this file created?
Hi Greg,
Quoted 1 line
From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
in linux-next. I think this patch is created on top of that by Moritz.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
"Note: I've seen that Mauro touched
Documentation/fpga/dfl.rstin linux-next commitc220a1fae6c5d("docs: fpga: convert docs to ReST and rename to *.rst")"Ok, but I can't take this patch just because the file is converted in someone else's tree :(
Oh.. if that patch is merged from fpga tree, then we wont have this problem. :(
You better not be basing your tree on linux-next :)
Quoted 2 lines
So do you have any suggestions on what should i do now? wait that patch goes to offical release, and then resubmit this patch?
Wait until the change hits Linus's tree and then resend it to me.
thanks,
greg k-h
Re: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
Quoted 33 lines · Greg KH
Quoted 32 lines · Wu Hao
Quoted 30 lines · Greg KH
Quoted 19 lines · Wu Hao
Quoted 16 lines · Greg KH
Quoted 6 lines · Moritz Fischer
From: Wu Hao <[email protected]>
This patch adds virtualization support description for DFL based FPGA devices (based on PCIe SRIOV), and introductions to new interfaces added by new dfl private feature drivers.
Quoted 8 lines
[[email protected]: Fixed up to make it work with new reStructuredText docs]Signed-off-by Xu Yilun <[email protected]>Signed-off-by Wu Hao <[email protected]>Acked-by Alan Tull <[email protected]>Signed-off-by Moritz Fischer <[email protected]>--- Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)Quoted 1 line
This doesn't apply to my tree, where is this file created?
Quoted 7 lines
Hi Greg,
Quoted 1 line
From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
in linux-next. I think this patch is created on top of that by Moritz.
"Note: I've seen that Mauro touched
Documentation/fpga/dfl.rstin linux-next commitc220a1fae6c5d("docs: fpga: convert docs to ReST and rename to *.rst")"Ok, but I can't take this patch just because the file is converted in someone else's tree :(
Quoted 8 lines
Quoted 1 line
Oh.. if that patch is merged from fpga tree, then we wont have this problem. :(
You better not be basing your tree on linux-next :)
Quoted 2 lines
So do you have any suggestions on what should i do now? wait that patch goes to offical release, and then resubmit this patch?
Wait until the change hits Linus's tree and then resend it to me.
OK, get it! Thanks!
Hao
Quoted 4 lines
thanks,
greg k-h
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
In reply to Wu HaoQuoted 36 lines · Wu Hao
Quoted 35 lines · Greg KH
Quoted 33 lines · Wu Hao
Quoted 18 lines · Greg KH
Quoted 17 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASEioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke thisPORT_ASSIGNioctl to attach the port back to PF.Ioctl interfaces:
*DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.Quoted 8 lines
Quoted 4 lines
*
DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.Why are you not using the "generic" bind/unbind facility that userspace already has for this with binding drivers to devices? Why a special ioctl?
Hi Greg,
Actually we think it should be safer that making the device invisble than just unbinding its driver. Looks like user can try to rebind it at any time and we don't have any method to stop them.
Quoted 8 lines
Quoted 3 lines
Why do you want to "stop" the user from doing something? They asked to do it, why prevent it? If they ask to do something foolish, well, they get to keep the pieces :)
Actually this is for SRIOV support, as we are moving FPGA accelerator from PF to VF, so we don't want users to see the FPGA accelerator from PF any more. We can't allow user to touch same FPGA accelerator from both PF and VF side (it leads to hardware erros).
Ick, ok, this needs to be documented really well then.
Quoted 79 lines
Quoted 39 lines
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
include/uapi/linux/fpga-dfl.h +10
-0@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release) +* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */ Quoted 8 lines
include/uapi/linux/fpga-dfl.h struct dfl_fpga_fme_port_pr { · line 191
Quoted 3 lines
include/uapi/linux/fpga-dfl.h +3
-0+__u32 flags;/* Zero for now */ +__u32 port_id; +};meta-comment, why do all of your structures for ioctls have argsz? You "know" the size of the structure already, it's part of the ioctl definition. You shouldn't need to also set it again, right? Otherwise ALL Linux ioctls would need something crazy like this.
Quoted 1 line
Actually we followed the same method as vfio.
vfio is a protocol on "the wire", right? Not an ioctl.
Quoted 2 lines
The major purpose should be extendibility, as we really need this to be sth long term maintainable.
You can't change ioctl structure sizes at any time.
Quoted 2 lines
It really helps, if we add some new members for extentions/enhancement under the same ioctl.
You don't do that.
Quoted 8 lines
Quoted 4 lines
I don't think everybody needs this, but my consideration here is if newer generations of hardware/specs come with some extentions, I still hope we can resue these IOCTLs as much as we could, instead of creating more new ones.
You create new ones, like everyone else does, as you can not change old code. By trying to "version" structures like this, it's just going to be a nightmare.
Actually i learned this from vfio code here, it's not trying to "version" structures, let me copy the comments from vfio header file. It should be more clear than above short description from me.
"
include/uapi/linux/vfio.h"/*
* The IOCTL interface is designed for extensibility by embedding the
* structure length (argsz) and flags into structures passed between
* kernel and userspace. We therefore use the_IO()macro for these
* defines to avoid implicitly embedding a size into the ioctl request.
* As structure fields are added, argsz will increase to match and flag
* bits will be defined to indicate additional fields with valid data.
* It's *always* the caller's responsibility to indicate the size of
* the structure passed by setting argsz appropriately.
*/For example.
struct vfio_device_info { __u32 argsz; __u32 flags; #define VFIO_DEVICE_FLAGS_RESET (1 << 0) /* Device supports reset */ #define VFIO_DEVICE_FLAGS_PCI (1 << 1) /* vfio-pci device */ #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */ #define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */ #define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */ #define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */ __u32 num_regions; /* Max region index + 1 */ __u32 num_irqs; /* Max IRQ index + 1 */
Quoted 1 line
Hope things could be more clear now. :)
That's nice for the vfio stuff, but you are just a "normal" driver here. You want an ioctl that just does one thing, no arguments, no flags, no anything. No need for a size argument then at all. These ioctls don't even need a structure for them!
Don't try to be fancy, it's not needed, it's not like you are running out of ioctl space...
thanks,
greg k-h
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
Quoted 41 lines · Greg KH
Quoted 40 lines · Wu Hao
Quoted 38 lines · Greg KH
Quoted 28 lines · Wu Hao
Quoted 18 lines · Greg KH
Quoted 17 lines · Moritz Fischer
From: Wu Hao <[email protected]>
In order to support virtualization usage via PCIe SRIOV, this patch adds two ioctls under FPGA Management Engine (FME) to release and assign back the port device. In order to safely turn Port from PF into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASEioctl to release port firstly to remove userspace interfaces, and then configure the PF/VF access register in FME. After disable SRIOV, it requires user to invoke thisPORT_ASSIGNioctl to attach the port back to PF.Ioctl interfaces:
*DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform device to remove related userspace interfaces on PF, then configures PF/VF access mode to VF.Quoted 8 lines
Quoted 4 lines
*
DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF, it configures PF/VF access mode to PF, then adds port platform device back to re-enable related userspace interfaces on PF.Why are you not using the "generic" bind/unbind facility that userspace already has for this with binding drivers to devices? Why a special ioctl?
Quoted 5 lines
Hi Greg,
Actually we think it should be safer that making the device invisble than just unbinding its driver. Looks like user can try to rebind it at any time and we don't have any method to stop them.
Why do you want to "stop" the user from doing something? They asked to do it, why prevent it? If they ask to do something foolish, well, they get to keep the pieces :)
Quoted 6 lines
Quoted 4 lines
Actually this is for SRIOV support, as we are moving FPGA accelerator from PF to VF, so we don't want users to see the FPGA accelerator from PF any more. We can't allow user to touch same FPGA accelerator from both PF and VF side (it leads to hardware erros).
Ick, ok, this needs to be documented really well then.
Yes, we have add relateded descriptions for virtualization support in that documentation patch.
[PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
Quoted 82 lines
Quoted 80 lines
Quoted 40 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
include/uapi/linux/fpga-dfl.h +10
-0@@ -176,4 +176,36 @@ struct dfl_fpga_fme_port_pr {#define DFL_FPGA_FME_PORT_PR _IO(DFL_FPGA_MAGIC, DFL_FME_BASE + 0) +/** +* DFL_FPGA_FME_PORT_RELEASE - _IOW(DFL_FPGA_MAGIC, DFL_FME_BASE + 1, +* struct dfl_fpga_fme_port_release) +* +* Driver releases the port per Port ID provided by caller. +* Return: 0 on success, -errno on failure. +*/ +struct dfl_fpga_fme_port_release { +/* Input */ +__u32 argsz;/* Structure length */ Quoted 8 lines
include/uapi/linux/fpga-dfl.h struct dfl_fpga_fme_port_pr { · line 191
Quoted 3 lines
include/uapi/linux/fpga-dfl.h +3
-0+__u32 flags;/* Zero for now */ +__u32 port_id; +};meta-comment, why do all of your structures for ioctls have argsz? You "know" the size of the structure already, it's part of the ioctl definition. You shouldn't need to also set it again, right? Otherwise ALL Linux ioctls would need something crazy like this.
Quoted 1 line
Actually we followed the same method as vfio.
vfio is a protocol on "the wire", right? Not an ioctl.
Quoted 2 lines
The major purpose should be extendibility, as we really need this to be sth long term maintainable.
You can't change ioctl structure sizes at any time.
Quoted 2 lines
It really helps, if we add some new members for extentions/enhancement under the same ioctl.
You don't do that.
Quoted 8 lines
Quoted 4 lines
I don't think everybody needs this, but my consideration here is if newer generations of hardware/specs come with some extentions, I still hope we can resue these IOCTLs as much as we could, instead of creating more new ones.
You create new ones, like everyone else does, as you can not change old code. By trying to "version" structures like this, it's just going to be a nightmare.
Actually i learned this from vfio code here, it's not trying to "version" structures, let me copy the comments from vfio header file. It should be more clear than above short description from me.
"
include/uapi/linux/vfio.h"/*
* The IOCTL interface is designed for extensibility by embedding the
* structure length (argsz) and flags into structures passed between
* kernel and userspace. We therefore use the_IO()macro for these
* defines to avoid implicitly embedding a size into the ioctl request.
* As structure fields are added, argsz will increase to match and flag
* bits will be defined to indicate additional fields with valid data.
* It's *always* the caller's responsibility to indicate the size of
* the structure passed by setting argsz appropriately.
*/For example.
struct vfio_device_info { __u32 argsz; __u32 flags; #define VFIO_DEVICE_FLAGS_RESET (1 << 0) /* Device supports reset */ #define VFIO_DEVICE_FLAGS_PCI (1 << 1) /* vfio-pci device */ #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */ #define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */ #define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */ #define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */ __u32 num_regions; /* Max region index + 1 */ __u32 num_irqs; /* Max IRQ index + 1 */Quoted 1 line
Hope things could be more clear now. :)
Quoted 7 lines
That's nice for the vfio stuff, but you are just a "normal" driver here. You want an ioctl that just does one thing, no arguments, no flags, no anything. No need for a size argument then at all. These ioctls don't even need a structure for them!
Don't try to be fancy, it's not needed, it's not like you are running out of ioctl space...
Thanks a lot for the comments and suggestions.
That's true, it's a "normal" driver, maybe I overly considered the extensibility of it. OK, Let me rework this patch to remove argsz from these two ioctls.
What about the existing ioctls for this driver, they have argsz too. shall I prepare another patch to remove them as well?
Hao
Quoted 4 lines
thanks,
greg k-h
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
Quoted 10 lines · Wu Hao
Quoted 9 lines
Quoted 1 line
Hope things could be more clear now. :)
That's nice for the vfio stuff, but you are just a "normal" driver here. You want an ioctl that just does one thing, no arguments, no flags, no anything. No need for a size argument then at all. These ioctls don't even need a structure for them!
Don't try to be fancy, it's not needed, it's not like you are running out of ioctl space...
Quoted 8 lines
Thanks a lot for the comments and suggestions.
That's true, it's a "normal" driver, maybe I overly considered the extensibility of it. OK, Let me rework this patch to remove argsz from these two ioctls.
What about the existing ioctls for this driver, they have argsz too. shall I prepare another patch to remove them as well?
I am hoping you actually have users for those ioctls in userspace today? If not, and no one is using them, then yes, please fix those too.
thanks,
greg k-h
Re: [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support.
Quoted 17 lines · Greg KH
Quoted 16 lines · Wu Hao
Quoted 9 lines
Quoted 1 line
Hope things could be more clear now. :)
That's nice for the vfio stuff, but you are just a "normal" driver here. You want an ioctl that just does one thing, no arguments, no flags, no anything. No need for a size argument then at all. These ioctls don't even need a structure for them!
Don't try to be fancy, it's not needed, it's not like you are running out of ioctl space...
Thanks a lot for the comments and suggestions.
That's true, it's a "normal" driver, maybe I overly considered the extensibility of it. OK, Let me rework this patch to remove argsz from these two ioctls.
Quoted 5 lines
Quoted 2 lines
What about the existing ioctls for this driver, they have argsz too. shall I prepare another patch to remove them as well?
I am hoping you actually have users for those ioctls in userspace today? If not, and no one is using them, then yes, please fix those too.
Yes, we have a few users, not many, e.g. https://github.com/OPAE/opae-sdk
I believe we may have more users as we are submitting code to make this driver more usable.
Let me think about this, if we want to do this clean up, we have to increase the API version to tell everybody, things are changed. If finally we decide to do this clean up, that will be a new patch after this patchset.
Many Thanks for your patient guide and suggestions. :)
Hao
Quoted 4 lines
thanks,
greg k-h