octeontx2-vf: Add network driver for virtual function
Subsystems (4)
v3 cover octeontx2-vf: Add network driver for virtual function
From: Sunil Goutham <[email protected]>
This patch series adds network driver for the virtual functions of OcteonTX2 SOC's resource virtualization unit (RVU).
Changes from v2:
* Removed Copyright license text.
* Removed wrapper fn()s around mutex_lock and unlock.
* Got rid of using macro with 'return'.
* Removed __weak fn()s.
- Sugested by Leon Romanovsky and Andrew Lunn
Changes from v1:
* Removed driver version and fixed authorship
* Removed driver version and fixed authorship in the already upstreamed AF, PF drivers.
* Removed unnecessary checks in sriov_enable and xmit fn()s.
* Removed WQ_MEM_RECLAIM flag while creating workqueue.
* Added lock in tx_timeout task.
* Added 'supported_coalesce_params' in ethtool ops.
* Minor other cleanups.
- Sugested by Jakub Kicinski
Geetha sowjanya (2):
octeontx2-pf: Handle VF function level reset
octeontx2-pf: Cleanup all receive buffers in SG descriptor
Sunil Goutham (3):
octeontx2-pf: Enable SRIOV and added VF mbox handling
octeontx2-af: Remove driver version and fix authorship
octeontx2-pf: Remove wrapper APIs for mutex lock and unlock
Tomasz Duszynski (3):
octeontx2-vf: Virtual function driver support
octeontx2-vf: Ethtool support
octeontx2-vf: Link event notification support
drivers/net/ethernet/marvell/octeontx2/Kconfig | 6 + drivers/net/ethernet/marvell/octeontx2/af/rvu.c | 4 +- .../net/ethernet/marvell/octeontx2/nic/Makefile | 2 + .../ethernet/marvell/octeontx2/nic/otx2_common.c | 99 +-- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 42 +- .../ethernet/marvell/octeontx2/nic/otx2_ethtool.c | 128 +++- .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 824 ++++++++++++++++++++- .../net/ethernet/marvell/octeontx2/nic/otx2_reg.h | 13 + .../net/ethernet/marvell/octeontx2/nic/otx2_txrx.c | 41 +- .../net/ethernet/marvell/octeontx2/nic/otx2_vf.c | 650 ++++++++++++++++ 10 files changed, 1697 insertions(+), 112 deletions(-) create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
Signature · 1 line
2.7.4
PATCH v3 1/8 octeontx2-pf: Enable SRIOV and added VF mbox handling
From: Sunil Goutham <[email protected]>
Added 'sriov_configure' to enable/disable virtual functions (VFs). Also added handling of mailbox messages from these VFs.
Admin function (AF) is the only one with all priviliges to configure HW, alloc resources etc etc, PFs and it's VFs have to request AF via mbox for all their needs. But unlike PFs, their VFs cannot send a mbox request directly. A VF shares a mailbox region with it's parent PF, so VF sends a mailbox msg to PF and then PF forwards it to AF. Then AF after processing sends response to PF which it again forwards to VF.
This patch adds support for this 'VF <=> PF <=> AF' mailbox communication.
--- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 3 + .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 443 +++++++++++++++++++++ 2 files changed, 446 insertions(+)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +3 -0
@@ -215,8 +215,11 @@ struct otx2_nic {/* Mbox */ struct mbox mbox; +struct mbox *mbox_pfvf;struct workqueue_struct *mbox_wq; +struct workqueue_struct *mbox_pfvf_wq; +u8 total_vfs;u16 pcifunc;/* RVU PF_FUNC */ u16 bpid[NIX_MAX_BPID_CHAN ];struct cgx_link_user_info linfo;
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +443 -0
479 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 2/8 octeontx2-pf: Handle VF function level reset
In reply to [email protected]From: Geetha sowjanya <[email protected]>
When FLR is initiated for a VF (PCI function level reset), the parent PF gets a interrupt. PF then sends a message to admin function (AF), which then cleanups all resources attached to that VF.
Also handled IRQs triggered when master enable bit is cleared or set for VFs. This handler just clears the transaction pending ie TRPEND bit.
--- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 7 + .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 235 ++++++++++++++++++++- 2 files changed, 239 insertions(+), 3 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +7 -0
@@ -191,6 +191,11 @@ struct otx2_hw {u64 cgx_tx_stats[CGX_TX_STATS_COUNT ]; }; +struct flr_work { +struct work_struct work; +struct otx2_nic *pf; +}; +struct refill_work {struct delayed_work pool_refill_work;struct otx2_nic *pf;
@@ -226,6 +231,8 @@ struct otx2_nic {u64 reset_count;struct work_struct reset_task; +struct workqueue_struct *flr_wq; +struct flr_work *flr_wrk;struct refill_work *refill_wrk;/* Ethtool stuff */
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +232 -3
269 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 3/8 octeontx2-vf: Virtual function driver support
In reply to [email protected]From: Tomasz Duszynski <[email protected]>
On OcteonTx2 silicon there two two types VFs, VFs that share the physical link with their parent SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK). Except for the underlying Rx/Tx channel mapping from netdev functionality perspective they are almost identical. This patch adds netdev driver support for these VFs.
Unlike it's parent PF a VF cannot directly communicate with admin function (AF) and it has to go through PF for the same. The mailbox communication with AF works like 'VF <=> PF <=> AF'.
Also functionality wise VF and PF are identical, hence to avoid code duplication PF driver's APIs are resued here for HW initialization, packet handling etc etc ie almost everything. For VF driver to compile as module exported few of the existing PF driver APIs.
--- drivers/net/ethernet/marvell/octeontx2/Kconfig | 6 + .../net/ethernet/marvell/octeontx2/nic/Makefile | 2 + .../ethernet/marvell/octeontx2/nic/otx2_common.c | 14 + .../ethernet/marvell/octeontx2/nic/otx2_common.h | 9 + .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 11 +- .../net/ethernet/marvell/octeontx2/nic/otx2_reg.h | 13 + .../net/ethernet/marvell/octeontx2/nic/otx2_txrx.c | 1 + .../net/ethernet/marvell/octeontx2/nic/otx2_vf.c | 646 +++++++++++++++++++++ 8 files changed, 701 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
drivers/net/ethernet/marvell/octeontx2/Kconfig +6 -0
@@ -33,3 +33,9 @@ config OCTEONTX2_PF depends on PCI help This driver supports Marvell's OcteonTX2 NIC physical function. + +config OCTEONTX2_VF + tristate "Marvell OcteonTX2 NIC Virtual Function driver" + depends on OCTEONTX2_PF + help + This driver supports Marvell's OcteonTX2 NIC virtual function.
drivers/net/ethernet/marvell/octeontx2/nic/Makefile +2 -0
@@ -4,7 +4,9 @@ # obj-$(CONFIG_OCTEONTX2_PF) += octeontx2_nicpf.o +obj-$(CONFIG_OCTEONTX2_VF) += octeontx2_nicvf.o octeontx2_nicpf-y := otx2_pf.o otx2_common.o otx2_txrx.o otx2_ethtool.o +octeontx2_nicvf-y := otx2_vf.o ccflags-y += -I$(srctree)/drivers/net/ethernet/marvell/octeontx2/af
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c +14 -0
@@ -128,6 +128,7 @@ void otx2_get_stats64(struct net_device *netdev, stats->tx_packets = dev_stats->tx_frames; stats->tx_dropped = dev_stats->tx_drops; } +EXPORT_SYMBOL (otx2_get_stats64);/* Sync MAC address with RVU AF */ static int otx2_hw_set_mac_addr (struct otx2_nic *pfvf,u8 *mac)
@@ -197,6 +198,7 @@ int otx2_set_mac_address(struct net_device *netdev, void *p)return 0 ; } +EXPORT_SYMBOL (otx2_set_mac_address);int otx2_hw_set_mtu (struct otx2_nic *pfvf,int mtu) {
@@ -225,6 +227,9 @@ int otx2_config_pause_frm(struct otx2_nic *pfvf)struct cgx_pause_frm_cfg *req;int err; +if (is_otx2_lbkvf (pfvf->pdev)) +return 0 ; +otx2_mbox_lock (&pfvf->mbox); req =otx2_mbox_alloc_msg_cgx_cfg_pause_frm (&pfvf->mbox);if (!req) {
@@ -416,6 +421,7 @@ void otx2_tx_timeout(struct net_device *netdev, unsigned int txq)schedule_work (&pfvf->reset_task); } +EXPORT_SYMBOL (otx2_tx_timeout);void otx2_get_mac_from_af (struct net_device *netdev) {
@@ -430,6 +436,7 @@ void otx2_get_mac_from_af(struct net_device *netdev)if (!is_valid_ether_addr (netdev->dev_addr))eth_hw_addr_random (netdev); } +EXPORT_SYMBOL (otx2_get_mac_from_af);static int otx2_get_link (struct otx2_nic *pfvf) {
@@ -1263,6 +1270,7 @@ int otx2_detach_resources(struct mbox *mbox)otx2_mbox_unlock (mbox);return 0 ; } +EXPORT_SYMBOL (otx2_detach_resources);int otx2_attach_npa_nix (struct otx2_nic *pfvf) {
@@ -1319,6 +1327,7 @@ int otx2_attach_npa_nix(struct otx2_nic *pfvf)return 0 ; } +EXPORT_SYMBOL (otx2_attach_npa_nix);void otx2_ctx_disable (struct mbox *mbox,int type,bool npa) {
@@ -1387,6 +1396,7 @@ void mbox_handler_nix_txsch_alloc(struct otx2_nic *pf, pf->hw.txschq_list[lvl][schq] = rsp->schq_list[lvl][schq]; } +EXPORT_SYMBOL (mbox_handler_nix_txsch_alloc);void mbox_handler_npa_lf_alloc (struct otx2_nic *pfvf,struct npa_lf_alloc_rsp *rsp)
@@ -1394,6 +1404,7 @@ void mbox_handler_npa_lf_alloc(struct otx2_nic *pfvf, pfvf->hw.stack_pg_ptrs = rsp->stack_pg_ptrs; pfvf->hw.stack_pg_bytes = rsp->stack_pg_bytes; } +EXPORT_SYMBOL (mbox_handler_npa_lf_alloc);void mbox_handler_nix_lf_alloc (struct otx2_nic *pfvf,struct nix_lf_alloc_rsp *rsp)
@@ -1404,6 +1415,7 @@ void mbox_handler_nix_lf_alloc(struct otx2_nic *pfvf, pfvf->hw.lso_tsov4_idx = rsp->lso_tsov4_idx; pfvf->hw.lso_tsov6_idx = rsp->lso_tsov6_idx; } +EXPORT_SYMBOL (mbox_handler_nix_lf_alloc);void mbox_handler_msix_offset (struct otx2_nic *pfvf,struct msix_offset_rsp *rsp)
@@ -1411,6 +1423,7 @@ void mbox_handler_msix_offset(struct otx2_nic *pfvf, pfvf->hw.npa_msixoff = rsp->npa_msixoff; pfvf->hw.nix_msixoff = rsp->nix_msixoff; } +EXPORT_SYMBOL (mbox_handler_msix_offset);void mbox_handler_nix_bp_enable (struct otx2_nic *pfvf,struct nix_bp_cfg_rsp *rsp)
@@ -1422,6 +1435,7 @@ void mbox_handler_nix_bp_enable(struct otx2_nic *pfvf, pfvf->bpid[chan_id] = rsp->chan_bpid[chan] &0x3FF ; } } +EXPORT_SYMBOL (mbox_handler_nix_bp_enable);void otx2_free_cints (struct otx2_nic *pfvf,int n) {
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +9 -0
@@ -20,6 +20,8 @@/* PCI device IDs */ #define PCI_DEVID_OCTEONTX2_RVU_PF 0xA063 +#define PCI_DEVID_OCTEONTX2_RVU_VF 0xA064 +#define PCI_DEVID_OCTEONTX2_RVU_AFVF 0xA0F8#define PCI_SUBSYS_DEVID_96XX_RVU_PFVF 0xB200
@@ -242,6 +244,11 @@ struct otx2_nic {int nix_blkaddr; }; +static inline bool is_otx2_lbkvf (struct pci_dev *pdev) +{ +return pdev->device ==PCI_DEVID_OCTEONTX2_RVU_AFVF ; +} +static inline bool is_96xx_A0 (struct pci_dev *pdev) { return (pdev->revision == 0x00) &&
@@ -627,6 +634,8 @@ void otx2_set_ethtool_ops(struct net_device *netdev);int otx2_open (struct net_device *netdev);int otx2_stop (struct net_device *netdev); +int otx2vf_open (struct net_device *netdev); +int otx2vf_stop (struct net_device *netdev);int otx2_set_real_num_queues (struct net_device *netdev,int tx_queues,int rx_queues);#endif /* OTX2_COMMON_H */
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +10 -1
@@ -1085,6 +1085,7 @@ int otx2_set_real_num_queues(struct net_device *netdev, "Failed to set no of Rx queues: %d\n", rx_queues);return err; } +EXPORT_SYMBOL (otx2_set_real_num_queues);static irqreturn_t otx2_q_intr_handler (int irq,void *data) {
@@ -1520,6 +1521,9 @@ int otx2_open(struct net_device *netdev)if (pf->linfo.link_up && !(pf->pcifunc &RVU_PFVF_FUNC_MASK ))otx2_handle_link_event (pf); +/* Restore pause frame settings */ +otx2_config_pause_frm (pf); + err =otx2_rxtx_enable (pf, true);if (err) goto err_free_cints;
@@ -1543,6 +1547,7 @@ int otx2_open(struct net_device *netdev)kfree (qset->napi);return err; } +EXPORT_SYMBOL (otx2_open);int otx2_stop (struct net_device *netdev) {
@@ -1603,6 +1608,7 @@ int otx2_stop(struct net_device *netdev) sizeof(*qset) - offsetof(struct otx2_qset, sqe_cnt));return 0 ; } +EXPORT_SYMBOL (otx2_stop);static netdev_tx_t otx2_xmit (struct sk_buff *skb,struct net_device *netdev) {
@@ -1731,7 +1737,6 @@ static int otx2_realloc_msix_vectors(struct otx2_nic *pf)otx2_disable_mbox_intr (pf);pci_free_irq_vectors (hw->pdev); -pci_free_irq_vectors (hw->pdev); err =pci_alloc_irq_vectors (hw->pdev, num_vec, num_vec,PCI_IRQ_MSIX );if (err <0 ) { dev_err(pf->dev, "%s: Failed to realloc %d IRQ vectors\n",
@@ -1898,6 +1903,10 @@ static int otx2_probe(struct pci_dev *pdev, const struct pci_device_id *id)/* Enable link notifications */ otx2_cgx_config_linkevents (pf, true); +/* Enable pause frames by default */ + pf->flags |=OTX2_FLAG_RX_PAUSE_ENABLED ; + pf->flags |=OTX2_FLAG_TX_PAUSE_ENABLED ; +return 0 ; err_detach_rsrc:
drivers/net/ethernet/marvell/octeontx2/nic/otx2_reg.h +13 -0
@@ -45,6 +45,19 @@#define RVU_PF_MSIX_VECX_CTL (a) (0x008 | (a) << 4)#define RVU_PF_MSIX_PBAX (a) (0xF0000 | (a) << 3) +/* RVU VF registers */ +#define RVU_VF_VFPF_MBOX0 (0x00000) +#define RVU_VF_VFPF_MBOX1 (0x00008) +#define RVU_VF_VFPF_MBOXX (a) (0x00 | (a) << 3) +#define RVU_VF_INT (0x20) +#define RVU_VF_INT_W1S (0x28) +#define RVU_VF_INT_ENA_W1S (0x30) +#define RVU_VF_INT_ENA_W1C (0x38) +#define RVU_VF_BLOCK_ADDRX_DISC (a) (0x200 | (a) << 3) +#define RVU_VF_MSIX_VECX_ADDR (a) (0x000 | (a) << 4) +#define RVU_VF_MSIX_VECX_CTL (a) (0x008 | (a) << 4) +#define RVU_VF_MSIX_PBAX (a) (0xF0000 | (a) << 3) +#define RVU_FUNC_BLKADDR_SHIFT 20#define RVU_FUNC_BLKADDR_MASK 0x1FULL
drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c +1 -0
@@ -778,6 +778,7 @@ bool otx2_sq_append_skb(struct net_device *netdev, struct otx2_snd_queue *sq,return true; } +EXPORT_SYMBOL (otx2_sq_append_skb);void otx2_cleanup_rx_cqes (struct otx2_nic *pfvf,struct otx2_cq_queue *cq) {
drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c new file +646 -0
646 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 4/8 octeontx2-vf: Ethtool support
In reply to [email protected]From: Tomasz Duszynski <[email protected]>
Added ethtool support for VF devices for
- Driver stats, Tx/Rx perqueue stats
- Set/show Rx/Tx queue count
- Set/show Rx/Tx ring sizes
- Set/show IRQ coalescing parameters
- RSS configuration etc
It's the PF which owns the interface, hence VF cannot display underlying CGX interface stats. Except for this rest ethtool support reuses PF's APIs.
--- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 1 + .../ethernet/marvell/octeontx2/nic/otx2_ethtool.c | 128 ++++++++++++++++++--- .../net/ethernet/marvell/octeontx2/nic/otx2_vf.c | 4 + 3 files changed, 119 insertions(+), 14 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +1 -0
@@ -631,6 +631,7 @@ void otx2_update_lmac_stats(struct otx2_nic *pfvf);int otx2_update_rq_stats (struct otx2_nic *pfvf,int qidx);int otx2_update_sq_stats (struct otx2_nic *pfvf,int qidx);void otx2_set_ethtool_ops (struct net_device *netdev); +void otx2vf_set_ethtool_ops (struct net_device *netdev);int otx2_open (struct net_device *netdev);int otx2_stop (struct net_device *netdev);
drivers/net/ethernet/marvell/octeontx2/nic/otx2_ethtool.c +114 -14
185 lines not on this page · raw message
drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c +4 -0
@@ -368,11 +368,13 @@ int otx2vf_open(struct net_device *netdev)return 0 ; } +EXPORT_SYMBOL (otx2vf_open);int otx2vf_stop (struct net_device *netdev) {return otx2_stop (netdev); } +EXPORT_SYMBOL (otx2vf_stop);static netdev_tx_t otx2vf_xmit (struct sk_buff *skb,struct net_device *netdev) {
@@ -579,6 +581,8 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id) goto err_detach_rsrc; } +otx2vf_set_ethtool_ops (netdev); +/* Enable pause frames by default */ vf->flags |=OTX2_FLAG_RX_PAUSE_ENABLED ; vf->flags |=OTX2_FLAG_TX_PAUSE_ENABLED ;
Signature · 1 line
2.7.4
Re: [PATCH v3 net-next 4/8] octeontx2-vf: Ethtool support
Quoted 9 lines · On Thu, Mar 19, 2020 at 06:37:24PM +0530, [email protected] wrote:
From: Tomasz Duszynski <[email protected]>
Added ethtool support for VF devices for
- Driver stats, Tx/Rx perqueue stats
- Set/show Rx/Tx queue count
- Set/show Rx/Tx ring sizes
- Set/show IRQ coalescing parameters
- RSS configuration etc
Quoted 7 lines
It's the PF which owns the interface, hence VF cannot display underlying CGX interface stats. Except for this rest ethtool support reuses PF's APIs.
Signed-off-by Tomasz Duszynski <[email protected]>Signed-off-by Sunil Goutham <[email protected]>
Andrew
Jakub Kicinski
maintainerRe: [PATCH v3 net-next 4/8] octeontx2-vf: Ethtool support
Quoted 15 lines · On Thu, 19 Mar 2020 16:56:31 +0100 Andrew Lunn wrote:
Quoted 14 lines · On Thu, Mar 19, 2020 at 06:37:24PM +0530, [email protected] wrote:
From: Tomasz Duszynski <[email protected]>
Added ethtool support for VF devices for
- Driver stats, Tx/Rx perqueue stats
- Set/show Rx/Tx queue count
- Set/show Rx/Tx ring sizes
- Set/show IRQ coalescing parameters
- RSS configuration etcIt's the PF which owns the interface, hence VF cannot display underlying CGX interface stats. Except for this rest ethtool support reuses PF's APIs.
Quoted 4 lines
Quoted 2 lines
Signed-off-by Tomasz Duszynski <[email protected]>Signed-off-by Sunil Goutham <[email protected]>Reviewed-by Andrew Lunn <[email protected]>
But they didn't add static inlines, no? Don't the dependencies look strange?
VF depends on PF code, but ethtool code (part of PF) also needs symbols from the VF..
Are the kconfig options really needed? If the dependencies are circular, this will either not work well or have all the modules loaded together. So let's just have everything under one kconfig knob and not bother breaking it up at module boundaries..
Note that it's perfectly fine to have multiple drivers in one module (it used to be harder long time ago).
Re: [PATCH v3 net-next 4/8] octeontx2-vf: Ethtool support
Quoted 20 lines · Jakub Kicinski
Quoted 18 lines · On Thu, 19 Mar 2020 16:56:31 +0100 Andrew Lunn wrote:
Quoted 16 lines · On Thu, Mar 19, 2020 at 06:37:24PM +0530, [email protected] wrote:
From: Tomasz Duszynski <[email protected]>
Added ethtool support for VF devices for
- Driver stats, Tx/Rx perqueue stats
- Set/show Rx/Tx queue count
- Set/show Rx/Tx ring sizes
- Set/show IRQ coalescing parameters
- RSS configuration etcIt's the PF which owns the interface, hence VF cannot display underlying CGX interface stats. Except for this rest ethtool support reuses PF's APIs.
Signed-off-by Tomasz Duszynski <[email protected]>Signed-off-by Sunil Goutham <[email protected]>
Quoted 8 lines
Quoted 1 line
Reviewed-by Andrew Lunn <[email protected]>But they didn't add static inlines, no? Don't the dependencies look strange?
VF depends on PF code, but ethtool code (part of PF) also needs symbols from the VF..
ethtool code has no dependency on symbols from the VF driver. PF driver compiles and module loads fine without enabling VF driver. While getting rid of __weak fn()s i forgot to remove EXPORT symbols from VF driver.
I will remove and resubmit.
Thanks,
Sunil.
Jakub Kicinski
maintainerRe: [PATCH v3 net-next 4/8] octeontx2-vf: Ethtool support
Quoted 30 lines · On Fri, 20 Mar 2020 12:35:15 +0530 Sunil Kovvuri wrote:
Quoted 20 lines · Jakub Kicinski
Quoted 18 lines · On Thu, 19 Mar 2020 16:56:31 +0100 Andrew Lunn wrote:
Quoted 16 lines · On Thu, Mar 19, 2020 at 06:37:24PM +0530, [email protected] wrote:
From: Tomasz Duszynski <[email protected]>
Added ethtool support for VF devices for
- Driver stats, Tx/Rx perqueue stats
- Set/show Rx/Tx queue count
- Set/show Rx/Tx ring sizes
- Set/show IRQ coalescing parameters
- RSS configuration etcIt's the PF which owns the interface, hence VF cannot display underlying CGX interface stats. Except for this rest ethtool support reuses PF's APIs.
Signed-off-by Tomasz Duszynski <[email protected]>Signed-off-by Sunil Goutham <[email protected]>Quoted 8 lines
Quoted 1 line
Reviewed-by Andrew Lunn <[email protected]>But they didn't add static inlines, no? Don't the dependencies look strange?
VF depends on PF code, but ethtool code (part of PF) also needs symbols from the VF..
Quoted 5 lines
ethtool code has no dependency on symbols from the VF driver. PF driver compiles and module loads fine without enabling VF driver. While getting rid of
__weakfn()s i forgot to remove EXPORT symbols from VF driver.
I will remove and resubmit.
I see. I saw the exports and assumed it gets called. Thanks.
PATCH v3 5/8 octeontx2-vf: Link event notification support
In reply to [email protected]From: Tomasz Duszynski <[email protected]>
VF shares physical link with PF. Admin function (AF) sends notification to PF whenever a link change event happens. PF has to forward the same notification to each of the enabled VF.
PF traps START/STOP_RX messages sent by VF to AF to keep track of VF's enabled/disabled state.
--- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 7 ++ .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 88 +++++++++++++++++++++- 2 files changed, 93 insertions(+), 2 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +7 -0
@@ -193,6 +193,12 @@ struct otx2_hw {u64 cgx_tx_stats[CGX_TX_STATS_COUNT ]; }; +struct otx2_vf_config { +struct otx2_nic *pf; +struct delayed_work link_event_work; +bool intf_down;/* interface was either configured or not */ +}; +struct flr_work {struct work_struct work;struct otx2_nic *pf;
@@ -229,6 +235,7 @@ struct otx2_nic {u8 total_vfs;u16 pcifunc;/* RVU PF_FUNC */ u16 bpid[NIX_MAX_BPID_CHAN ]; +struct otx2_vf_config *vf_configs;struct cgx_link_user_info linfo;u64 reset_count;
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +86 -2
145 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 6/8 octeontx2-pf: Cleanup all receive buffers in SG descriptor
In reply to [email protected]From: Geetha sowjanya <[email protected]>
With MTU sized receive buffers it is not expected to have CQE_RX with multiple receive buffer pointers. But since same physcial link is shared by PF and it's VFs, the max receive packet configured at link could be morethan MTU. Hence there is a chance of receiving plts morethan MTU which then gets DMA'ed into multiple buffers and notified in a single CQE_RX. This patch treats such pkts as errors and frees up receive buffers pointers back to hardware.
Also on the transmit side this patch sets SMQ MAXLEN to max value to avoid HW length errors for the packets whose size > MTU, eg due to path MTU.
--- .../ethernet/marvell/octeontx2/nic/otx2_common.c | 9 +++--- .../net/ethernet/marvell/octeontx2/nic/otx2_txrx.c | 34 ++++++++++++++++++---- 2 files changed, 34 insertions(+), 9 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c +5 -4
@@ -212,8 +212,6 @@ int otx2_hw_set_mtu(struct otx2_nic *pfvf, int mtu)return -ENOMEM ; } -/* SMQ config limits maximum pkt size that can be transmitted */ - req->update_smq = true; pfvf->max_frs = mtu +OTX2_ETH_HLEN ; req->maxlen = pfvf->max_frs;
@@ -472,7 +470,7 @@ int otx2_txschq_config(struct otx2_nic *pfvf, int lvl)/* Set topology e.t.c configuration */ if (lvl ==NIX_TXSCH_LVL_SMQ ) { req->reg[0 ] =NIX_AF_SMQX_CFG (schq); - req->regval[0 ] = ((pfvf->netdev->mtu +OTX2_ETH_HLEN ) <<8 ) | + req->regval[0 ] = ((OTX2_MAX_MTU +OTX2_ETH_HLEN ) <<8 ) |OTX2_MIN_MTU ; req->regval[0] |= (0x20ULL << 51) | (0x80ULL << 39) |
@@ -582,17 +580,19 @@ void otx2_sqb_flush(struct otx2_nic *pfvf) {int qidx, sqe_tail, sqe_head;u64 incr, *ptr, val; +int timeout =1000 ; ptr = (u64 *)otx2_get_regaddr (pfvf,NIX_LF_SQ_OP_STATUS );for (qidx =0 ; qidx < pfvf->hw.tx_queues; qidx++) { incr = (u64 )qidx <<32 ; -while (1 ) { +while (timeout) { val =otx2_atomic64_add (incr, ptr); sqe_head = (val >>20 ) &0x3F ; sqe_tail = (val >>28 ) &0x3F ;if (sqe_head == sqe_tail)break ;usleep_range (1 ,3 ); + timeout--; } } }
@@ -988,6 +988,7 @@ void otx2_aura_pool_free(struct otx2_nic *pfvf)qmem_free (pfvf->dev, pool->fc_addr); }devm_kfree (pfvf->dev, pfvf->qset.pool); + pfvf->qset.pool =NULL ; } static int otx2_aura_init(struct otx2_nic *pfvf, int aura_id,
drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c +29 -5
@@ -138,6 +138,25 @@ static void otx2_set_rxhash(struct otx2_nic *pfvf,skb_set_hash (skb, hash, hash_type); } +static void otx2_free_rcv_seg (struct otx2_nic *pfvf,struct nix_cqe_rx_s *cqe, +int qidx) +{ +struct nix_rx_sg_s *sg = &cqe->sg; +void *end, *start; +u64 *seg_addr; +int seg; + + start = (void *)sg; + end = start + ((cqe->parse.desc_sizem1 +1 ) *16 ); +while (start < end) { + sg = (struct nix_rx_sg_s *)start; + seg_addr = &sg->seg_addr; +for (seg =0 ; seg < sg->segs; seg++, seg_addr++) +otx2_aura_freeptr (pfvf, qidx, *seg_addr & ~0x07ULL ); + start +=sizeof (*sg); + } +} +static bool otx2_check_rcv_errors (struct otx2_nic *pfvf,struct nix_cqe_rx_s *cqe,int qidx) {
@@ -189,16 +208,17 @@ static bool otx2_check_rcv_errors(struct otx2_nic *pfvf,/* For now ignore all the NPC parser errors and * pass the packets to stack. */ -return false; +if (cqe->sg.segs ==1 ) +return false; }/* If RXALL is enabled pass on packets to stack. */ -if (cqe->sg.segs && (pfvf->netdev->features &NETIF_F_RXALL )) +if (cqe->sg.segs ==1 && (pfvf->netdev->features &NETIF_F_RXALL ))return false;/* Free buffer back to pool */ if (cqe->sg.segs) -otx2_aura_freeptr (pfvf, qidx, cqe->sg.seg_addr & ~0x07ULL ); +otx2_free_rcv_seg (pfvf, cqe, qidx);return true; }
@@ -210,7 +230,7 @@ static void otx2_rcv_pkt_handler(struct otx2_nic *pfvf,struct nix_rx_parse_s *parse = &cqe->parse;struct sk_buff *skb =NULL ; -if (unlikely (parse->errlev || parse->errcode)) { +if (unlikely (parse->errlev || parse->errcode || cqe->sg.segs >1 )) {if (otx2_check_rcv_errors (pfvf, cqe, cq->cq_idx))return ; }
@@ -789,11 +809,15 @@ void otx2_cleanup_rx_cqes(struct otx2_nic *pfvf, struct otx2_cq_queue *cq)while ((cqe = (struct nix_cqe_rx_s *)otx2_get_next_cqe (cq))) {if (!cqe->sg.subdc)continue ; + processed_cqe++; +if (cqe->sg.segs >1 ) { +otx2_free_rcv_seg (pfvf, cqe, cq->cq_idx); +continue ; + } iova = cqe->sg.seg_addr -OTX2_HEAD_ROOM ; pa =otx2_iova_to_phys (pfvf->iommu_domain, iova);otx2_dma_unmap_page (pfvf, iova, pfvf->rbsize,DMA_FROM_DEVICE );put_page (virt_to_page (phys_to_virt (pa))); - processed_cqe++; }/* Free CQEs to HW */
Signature · 1 line
2.7.4
PATCH v3 7/8 octeontx2-af: Remove driver version and fix authorship
In reply to [email protected]From: Sunil Goutham <[email protected]>
Removed MODULE_VERSION and fixed MODULE_AUTHOR.
--- drivers/net/ethernet/marvell/octeontx2/af/rvu.c | 4 +--- drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-)
drivers/net/ethernet/marvell/octeontx2/af/rvu.c +1 -3
@@ -21,7 +21,6 @@#define DRV_NAME "octeontx2-af"#define DRV_STRING "Marvell OcteonTX2 RVU Admin Function Driver" -#define DRV_VERSION "1.0"static int rvu_get_hwvf (struct rvu *rvu,int pcifunc);
@@ -46,10 +45,9 @@ static const struct pci_device_id rvu_id_table[] = { { 0, } /* end of table */ }; -MODULE_AUTHOR ("Marvell International Ltd." ); +MODULE_AUTHOR ("Sunil Goutham <[email protected]>" );MODULE_DESCRIPTION (DRV_STRING );MODULE_LICENSE ("GPL v2" ); -MODULE_VERSION (DRV_VERSION );MODULE_DEVICE_TABLE (pci, rvu_id_table);static char *mkex_profile;/* MKEX profile name */
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +1 -3
@@ -24,7 +24,6 @@#define DRV_NAME "octeontx2-nicpf"#define DRV_STRING "Marvell OcteonTX2 NIC Physical Function Driver" -#define DRV_VERSION "1.0"/* Supported devices */ static const struct pci_device_id otx2_pf_id_table[] = {
@@ -32,10 +31,9 @@ static const struct pci_device_id otx2_pf_id_table[] = { { 0, } /* end of table */ }; -MODULE_AUTHOR ("Marvell International Ltd." ); +MODULE_AUTHOR ("Sunil Goutham <[email protected]>" );MODULE_DESCRIPTION (DRV_STRING );MODULE_LICENSE ("GPL v2" ); -MODULE_VERSION (DRV_VERSION );MODULE_DEVICE_TABLE (pci, otx2_pf_id_table);enum {
Signature · 1 line
2.7.4
PATCH v3 8/8 octeontx2-pf: Remove wrapper APIs for mutex lock and unlock
In reply to [email protected]From: Sunil Goutham <[email protected]>
This patch removes wrapper fn()s around mutex_init/lock/unlock.
--- .../ethernet/marvell/octeontx2/nic/otx2_common.c | 76 +++++++++++----------- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 15 ----- .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 65 +++++++++--------- .../net/ethernet/marvell/octeontx2/nic/otx2_txrx.c | 6 +- .../net/ethernet/marvell/octeontx2/nic/otx2_vf.c | 2 +- 5 files changed, 75 insertions(+), 89 deletions(-)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c +38 -38
@@ -49,15 +49,15 @@ void otx2_update_lmac_stats(struct otx2_nic *pfvf)if (!netif_running (pfvf->netdev))return ; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock); req =otx2_mbox_alloc_msg_cgx_stats (&pfvf->mbox);if (!req) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return ; }otx2_sync_mbox_msg (&pfvf->mbox); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock); }int otx2_update_rq_stats (struct otx2_nic *pfvf,int qidx)
@@ -136,17 +136,17 @@ static int otx2_hw_set_mac_addr(struct otx2_nic *pfvf, u8 *mac)struct nix_set_mac_addr *req;int err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock); req =otx2_mbox_alloc_msg_nix_set_mac_addr (&pfvf->mbox);if (!req) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; }ether_addr_copy (req->mac_addr, mac); err =otx2_sync_mbox_msg (&pfvf->mbox); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; }
@@ -158,27 +158,27 @@ static int otx2_hw_get_mac_addr(struct otx2_nic *pfvf,struct msg_req *req;int err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock); req =otx2_mbox_alloc_msg_nix_get_mac_addr (&pfvf->mbox);if (!req) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; } err =otx2_sync_mbox_msg (&pfvf->mbox);if (err) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; } msghdr =otx2_mbox_get_rsp (&pfvf->mbox.mbox,0 , &req->hdr);if (IS_ERR (msghdr)) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return PTR_ERR (msghdr); } rsp = (struct nix_get_mac_addr_rsp *)msghdr;ether_addr_copy (netdev->dev_addr, rsp->mac_addr); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return 0 ; }
@@ -205,10 +205,10 @@ int otx2_hw_set_mtu(struct otx2_nic *pfvf, int mtu)struct nix_frs_cfg *req;int err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock); req =otx2_mbox_alloc_msg_nix_set_hw_frs (&pfvf->mbox);if (!req) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; }
@@ -216,7 +216,7 @@ int otx2_hw_set_mtu(struct otx2_nic *pfvf, int mtu) req->maxlen = pfvf->max_frs; err =otx2_sync_mbox_msg (&pfvf->mbox); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; }
@@ -228,7 +228,7 @@ int otx2_config_pause_frm(struct otx2_nic *pfvf)if (is_otx2_lbkvf (pfvf->pdev))return 0 ; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock); req =otx2_mbox_alloc_msg_cgx_cfg_pause_frm (&pfvf->mbox);if (!req) { err = -ENOMEM ;
@@ -241,7 +241,7 @@ int otx2_config_pause_frm(struct otx2_nic *pfvf) err =otx2_sync_mbox_msg (&pfvf->mbox); unlock: -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; }
@@ -251,10 +251,10 @@ int otx2_set_flowkey_cfg(struct otx2_nic *pfvf)struct nix_rss_flowkey_cfg *req;int err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock); req =otx2_mbox_alloc_msg_nix_rss_flowkey_cfg (&pfvf->mbox);if (!req) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; } req->mcam_index =-1 ;/* Default or reserved index */
@@ -262,7 +262,7 @@ int otx2_set_flowkey_cfg(struct otx2_nic *pfvf) req->group =DEFAULT_RSS_CONTEXT_GROUP ; err =otx2_sync_mbox_msg (&pfvf->mbox); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; }
@@ -273,7 +273,7 @@ int otx2_set_rss_table(struct otx2_nic *pfvf)struct nix_aq_enq_req *aq;int idx, err; -otx2_mbox_lock (mbox); +mutex_lock (&mbox->lock);/* Get memory to put this msg */ for (idx =0 ; idx < rss->rss_size; idx++) { aq =otx2_mbox_alloc_msg_nix_aq_enq (mbox);
@@ -283,12 +283,12 @@ int otx2_set_rss_table(struct otx2_nic *pfvf)*/ err =otx2_sync_mbox_msg (mbox);if (err) { -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock);return err; } aq =otx2_mbox_alloc_msg_nix_aq_enq (mbox);if (!aq) { -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock);return -ENOMEM ; } }
@@ -301,7 +301,7 @@ int otx2_set_rss_table(struct otx2_nic *pfvf) aq->op =NIX_AQ_INSTOP_INIT ; } err =otx2_sync_mbox_msg (mbox); -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock);return err; }
@@ -556,17 +556,17 @@ int otx2_txschq_stop(struct otx2_nic *pfvf)struct nix_txsch_free_req *free_req;int lvl, schq, err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock);/* Free the transmit schedulers */ free_req =otx2_mbox_alloc_msg_nix_txsch_free (&pfvf->mbox);if (!free_req) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; } free_req->flags =TXSCHQ_FREE_ALL ; err =otx2_sync_mbox_msg (&pfvf->mbox); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);/* Clear the txschq list */ for (lvl =0 ; lvl <NIX_TXSCH_LVL_CNT ; lvl++) {
@@ -1256,10 +1256,10 @@ int otx2_detach_resources(struct mbox *mbox) {struct rsrc_detach *detach; -otx2_mbox_lock (mbox); +mutex_lock (&mbox->lock); detach =otx2_mbox_alloc_msg_detach_resources (mbox);if (!detach) { -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock);return -ENOMEM ; }
@@ -1268,7 +1268,7 @@ int otx2_detach_resources(struct mbox *mbox)/* Send detach request to AF */ otx2_mbox_msg_send (&mbox->mbox,0 ); -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock);return 0 ; }EXPORT_SYMBOL (otx2_detach_resources);
@@ -1279,11 +1279,11 @@ int otx2_attach_npa_nix(struct otx2_nic *pfvf)struct msg_req *msix;int err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock);/* Get memory to put this msg */ attach =otx2_mbox_alloc_msg_attach_resources (&pfvf->mbox);if (!attach) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; }
@@ -1293,7 +1293,7 @@ int otx2_attach_npa_nix(struct otx2_nic *pfvf)/* Send attach request to AF */ err =otx2_sync_mbox_msg (&pfvf->mbox);if (err) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; }
@@ -1308,16 +1308,16 @@ int otx2_attach_npa_nix(struct otx2_nic *pfvf)/* Get NPA and NIX MSIX vector offsets */ msix =otx2_mbox_alloc_msg_msix_offset (&pfvf->mbox);if (!msix) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; } err =otx2_sync_mbox_msg (&pfvf->mbox);if (err) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; } -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);if (pfvf->hw.npa_msixoff ==MSIX_VECTOR_INVALID || pfvf->hw.nix_msixoff ==MSIX_VECTOR_INVALID ) {
@@ -1334,7 +1334,7 @@ void otx2_ctx_disable(struct mbox *mbox, int type, bool npa) {struct hwctx_disable_req *req; -otx2_mbox_lock (mbox); +mutex_lock (&mbox->lock);/* Request AQ to disable this context */ if (npa) req =otx2_mbox_alloc_msg_npa_hwctx_disable (mbox);
@@ -1342,7 +1342,7 @@ void otx2_ctx_disable(struct mbox *mbox, int type, bool npa) req =otx2_mbox_alloc_msg_nix_hwctx_disable (mbox);if (!req) { -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock);return ; }
@@ -1352,7 +1352,7 @@ void otx2_ctx_disable(struct mbox *mbox, int type, bool npa)dev_err (mbox->pfvf->dev,"%s failed to disable context\n" , __func__); -otx2_mbox_unlock (mbox); +mutex_unlock (&mbox->lock); }int otx2_nix_config_bp (struct otx2_nic *pfvf,bool enable)
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +0 -15
@@ -375,21 +375,6 @@ static inline void otx2_sync_mbox_bbuf(struct otx2_mbox *mbox, int devid) hw_mbase + mbox->rx_start, msg_size + msgs_offset); } -static inline void otx2_mbox_lock_init (struct mbox *mbox) -{ -mutex_init (&mbox->lock); -} - -static inline void otx2_mbox_lock (struct mbox *mbox) -{ -mutex_lock (&mbox->lock); -} - -static inline void otx2_mbox_unlock (struct mbox *mbox) -{ -mutex_unlock (&mbox->lock); -} -/* With the absence of API for 128-bit IO memory access for arm64, * implement required operations at place. */
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +33 -32
233 lines not on this page · raw message
drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c +3 -3
@@ -856,18 +856,18 @@ int otx2_rxtx_enable(struct otx2_nic *pfvf, bool enable)struct msg_req *msg;int err; -otx2_mbox_lock (&pfvf->mbox); +mutex_lock (&pfvf->mbox.lock);if (enable) msg =otx2_mbox_alloc_msg_nix_lf_start_rx (&pfvf->mbox);else msg =otx2_mbox_alloc_msg_nix_lf_stop_rx (&pfvf->mbox);if (!msg) { -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return -ENOMEM ; } err =otx2_sync_mbox_msg (&pfvf->mbox); -otx2_mbox_unlock (&pfvf->mbox); +mutex_unlock (&pfvf->mbox.lock);return err; }
drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c +1 -1
@@ -341,7 +341,7 @@ static int otx2vf_vfaf_mbox_init(struct otx2_nic *vf)INIT_WORK (&mbox->mbox_wrk, otx2vf_vfaf_mbox_handler);INIT_WORK (&mbox->mbox_up_wrk, otx2vf_vfaf_mbox_up_handler); -otx2_mbox_lock_init (&vf->mbox); +mutex_init (&mbox->lock);return 0 ; exit:
Signature · 1 line
2.7.4