dmaengine: idxd: Add basic DSA 3.0 capability and SGL support
Subsystems (2)
v2 cover dmaengine: idxd: Add basic DSA 3.0 capability and SGL support
This patch series introduces foundational support for DSA 3.0 features, exposing hardware capability registers to userspace in the IDXD driver.
DSA 3.0 introduces several new features that require awareness and configuration from both kernel and userspace. It is necessary to understand the hardware's capabilities for userspace tools (e.g., idxd-config, libraries, and applications) to make use of the features properly, such as supported features, memory layouts, and opcode compatibility.
Patch 1/2 exposes the three new capability registers (dsacap0-2) introduced in the DSA 3.0 specification through a new sysfs entry. This allows tools and users to query hardware capabilities such as supported SGL formats, floating-point options, and maximum supported sizes.
Patch 2/2 enables configuration of the maximum SGL size for DSA 3.0 devices. Some DSA 3.0 opcodes (e.g., Gather Copy, Gather Reduce) require that the workqueue's SGL size is explicitly configured. This patch sets that value based on hardware capabilities at initialization time, allowing these opcodes to function without additional user configuration.
---
Changes in v2:
- Added the link to the DSA 3.0 spec in the commit message (Dave)
- Fixed typos in the commit messages (Fenghua)
- Updated the sysfs ABI documentation for accuracy (Fenghua)
- Renamed the ABI entry from 'dsacap' to 'dsacaps' (Fenghua, Philip)
- Moved the definition of dsacap0_reg from patch #2 to patch #1 (Fenghua)
- Fixed the output format (Fenghua, Philip)
- Reordered the capability registers to match the DSA 3.0 spec (Fenghua)
- Add conditon checking to avoid accessing dsacaps when DSA 3.0 is not supported (Fenghua)
Yi Sun (2):
dmaengine: idxd: Expose DSA3.0 capabilities through sysfs
dmaengine: idxd: Add Max SGL Size Support for DSA3.0
.../ABI/stable/sysfs-driver-dma-idxd | 15 ++++++++++ drivers/dma/idxd/device.c | 5 ++++ drivers/dma/idxd/idxd.h | 19 +++++++++++++ drivers/dma/idxd/init.c | 11 ++++++++ drivers/dma/idxd/registers.h | 28 ++++++++++++++++++- drivers/dma/idxd/sysfs.c | 24 ++++++++++++++++ 6 files changed, 101 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.0
PATCH v2 1/2 dmaengine: idxd: Expose DSA3.0 capabilities through sysfs
Introduce sysfs interfaces for 3 new Data Streaming Accelerator (DSA) capability registers (dsacap0-2) to enable userspace awareness of hardware features in DSA version 3 and later devices.
Userspace components (e.g. configure libraries, workload Apps) require this information to:
1. Select optimal data transfer strategies based on SGL capabilities
2. Enable hardware-specific optimizations for floating-point operations
3. Configure memory operations with proper numerical handling
4. Verify compute operation compatibility before submitting jobs
The output format is <dsacap2>,<dsacap1>,<dsacap0>, where each DSA capability value is a 64-bit hexadecimal number, separated by commas. The ordering follows the DSA 3.0 specification layout:
Offset: 0x190 0x188 0x180 Reg: dsacap2 dsacap1 dsacap0
Example:
cat /sys/bus/dsa/devices/dsa0/dsacaps
000000000000f18d,0014000e000007aa,00fa01ff01ff03ff
According to the DSA 3.0 specification, there are 15 fields defined for the three dsacap registers. However, there's no need to define all register structures unless a use case requires them. At this point, support for the Scatter-Gather List (SGL) located in dsacap0 is necessary, so only dsacap0 is defined accordingly.
For reference, the DSA 3.0 specification is available at:
Documentation/ABI/stable/sysfs-driver-dma-idxd +15 -0
@@ -136,6 +136,21 @@ Description: The last executed device administrative command's status/error. Also last configuration error overloaded. Writing to it will clear the status. +What: /sys/bus/dsa/devices/dsa<m>/dsacaps +Date: Oct 5, 2025 +KernelVersion: 6.17.0 +Contact: [email protected] +Description: The DSA3 specification introduces three new capability + registers: dsacap[0-2]. User components (e.g., configuration + libraries and workload applications) require this information + to properly utilize the DSA3 features. + This includes SGL capability support, Enabling hardware-specific + optimizations, Configuring memory, etc. + The output format is '<dsacap2>,<dsacap1>,<dsacap0>' where each + DSA cap value is a 64 bit hex value. + This attribute should only be visible on DSA devices of version + 3 or later. + What: /sys/bus/dsa/devices/dsa<m>/iaa_cap Date: Sept 14, 2022 KernelVersion: 6.0.0
drivers/dma/idxd/idxd.h +3 -0
@@ -252,6 +252,9 @@ struct idxd_hw {struct opcap opcap;u32 cmd_cap;union iaa_cap_reg iaa_cap; +union dsacap0_reg dsacap0; +union dsacap1_reg dsacap1; +union dsacap2_reg dsacap2; };enum idxd_device_state {
drivers/dma/idxd/init.c +6 -0
@@ -582,6 +582,12 @@ static void idxd_read_caps(struct idxd_device *idxd) }multi_u64_to_bmap (idxd->opcap_bmap, &idxd->hw.opcap.bits[0 ],4 ); +if (idxd->hw.version >=DEVICE_VERSION_3 ) { + idxd->hw.dsacap0.bits =ioread64 (idxd->reg_base +IDXD_DSACAP0_OFFSET ); + idxd->hw.dsacap1.bits =ioread64 (idxd->reg_base +IDXD_DSACAP1_OFFSET ); + idxd->hw.dsacap2.bits =ioread64 (idxd->reg_base +IDXD_DSACAP2_OFFSET ); + } +/* read iaa cap */ if (idxd->data->type ==IDXD_TYPE_IAX && idxd->hw.version >=DEVICE_VERSION_2 ) idxd->hw.iaa_cap.bits =ioread64 (idxd->reg_base +IDXD_IAACAP_OFFSET );
drivers/dma/idxd/registers.h +25 -0
@@ -13,6 +13,7 @@#define DEVICE_VERSION_1 0x100#define DEVICE_VERSION_2 0x200 +#define DEVICE_VERSION_3 0x300#define IDXD_MMIO_BAR 0#define IDXD_WQ_BAR 2
@@ -582,6 +583,30 @@ union evl_status_reg {u64 bits; } __packed; +#define IDXD_DSACAP0_OFFSET 0x180 +union dsacap0_reg { +u64 bits; +struct { +u64 max_sgl_shift:4 ; +u64 max_gr_block_shift:4 ; +u64 ops_inter_domain:7 ; +u64 rsvd1:17 ; +u64 sgl_formats:16 ; +u64 max_sg_process:8 ; +u64 rsvd2:8 ; + }; +}; + +#define IDXD_DSACAP1_OFFSET 0x188 +union dsacap1_reg { +u64 bits; +}; + +#define IDXD_DSACAP2_OFFSET 0x190 +union dsacap2_reg { +u64 bits; +}; +#define IDXD_MAX_BATCH_IDENT 256struct __evl_entry {
drivers/dma/idxd/sysfs.c +24 -0
@@ -1713,6 +1713,18 @@ static ssize_t event_log_size_store(struct device *dev, }static DEVICE_ATTR_RW (event_log_size); +static ssize_t dsacaps_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct idxd_device *idxd =confdev_to_idxd (dev); + +return sysfs_emit (buf,"%016llx,%016llx,%016llx\n" , + (u64 )idxd->hw.dsacap2.bits, + (u64 )idxd->hw.dsacap1.bits, + (u64 )idxd->hw.dsacap0.bits); +} +static DEVICE_ATTR_RO (dsacaps); +static bool idxd_device_attr_max_batch_size_invisible (struct attribute *attr,struct idxd_device *idxd) {
@@ -1750,6 +1762,14 @@ static bool idxd_device_attr_event_log_size_invisible(struct attribute *attr, !idxd->hw.gen_cap.evl_support); } +static bool idxd_device_attr_dsacaps_invisible (struct attribute *attr, +struct idxd_device *idxd) +{ +return attr == &dev_attr_dsacaps.attr && + (idxd->data->type !=IDXD_TYPE_DSA || + idxd->hw.version <DEVICE_VERSION_3 ); +} +static umode_t idxd_device_attr_visible (struct kobject *kobj,struct attribute *attr,int n) {
@@ -1768,6 +1788,9 @@ static umode_t idxd_device_attr_visible(struct kobject *kobj,if (idxd_device_attr_event_log_size_invisible (attr, idxd))return 0 ; +if (idxd_device_attr_dsacaps_invisible (attr, idxd)) +return 0 ; +return attr->mode; }
@@ -1795,6 +1818,7 @@ static struct attribute *idxd_device_attributes[] = { &dev_attr_cmd_status.attr, &dev_attr_iaa_cap.attr, &dev_attr_event_log_size.attr, + &dev_attr_dsacaps.attr, NULL, };
Signature · 1 line
2.43.0
Re: [PATCH v2 1/2] dmaengine: idxd: Expose DSA3.0 capabilities through sysfs
Quoted 27 lines · Yi Sun
Introduce sysfs interfaces for 3 new Data Streaming Accelerator (DSA) capability registers (dsacap0-2) to enable userspace awareness of hardware features in DSA version 3 and later devices.
Userspace components (e.g. configure libraries, workload Apps) require this information to:
1. Select optimal data transfer strategies based on SGL capabilities
2. Enable hardware-specific optimizations for floating-point operations
3. Configure memory operations with proper numerical handling
4. Verify compute operation compatibility before submitting jobsThe output format is <dsacap2>,<dsacap1>,<dsacap0>, where each DSA capability value is a 64-bit hexadecimal number, separated by commas. The ordering follows the DSA 3.0 specification layout:
Offset: 0x190 0x188 0x180 Reg: dsacap2 dsacap1 dsacap0Example:
cat/sys/bus/dsa/devices/dsa0/dsacaps
000000000000f18d,0014000e000007aa,00fa01ff01ff03ffAccording to the DSA 3.0 specification, there are 15 fields defined for the three dsacap registers. However, there's no need to define all register structures unless a use case requires them. At this point, support for the Scatter-Gather List (SGL) located in dsacap0 is necessary, so only dsacap0 is defined accordingly.
Quoted 8 lines
For reference, the DSA 3.0 specification is available at:
Signed-off-by Yi Sun <[email protected]>Co-developed-by Anil S Keshavamurthy <[email protected]>Signed-off-by Anil S Keshavamurthy <[email protected]>Reviewed-by Dave Jiang <[email protected]>
Cheers,
Signature · 1 line
Vinicius
PATCH v2 2/2 dmaengine: idxd: Add Max SGL Size Support for DSA3.0
In reply to Yi SunCertain DSA 3.0 opcodes, such as Gather copy and Gather reduce, require max SGL configured for workqueues prior to supporting these opcodes.
Configure the maximum scatter-gather list (SGL) size for workqueues during setup on the supported HW. Application can then properly handle the SGL size without explicitly setting it.
drivers/dma/idxd/device.c +5 -0
@@ -375,6 +375,7 @@ static void idxd_wq_disable_cleanup(struct idxd_wq *wq)memset (wq->name,0 ,WQ_NAME_SIZE ); wq->max_xfer_bytes =WQ_DEFAULT_MAX_XFER ;idxd_wq_set_max_batch_size (idxd->data->type, wq,WQ_DEFAULT_MAX_BATCH ); +idxd_wq_set_init_max_sgl_size (idxd, wq);if (wq->opcap_bmap)bitmap_copy (wq->opcap_bmap, idxd->opcap_bmap,IDXD_MAX_OPCAP_BITS ); }
@@ -974,6 +975,8 @@ static int idxd_wq_config_write(struct idxd_wq *wq)/* bytes 12-15 */ wq->wqcfg->max_xfer_shift =ilog2 (wq->max_xfer_bytes);idxd_wqcfg_set_max_batch_shift (idxd->data->type, wq->wqcfg,ilog2 (wq->max_batch_size)); +if (idxd_sgl_supported (idxd)) + wq->wqcfg->max_sgl_shift =ilog2 (wq->max_sgl_size);/* bytes 32-63 */ if (idxd->hw.wq_cap.op_config && wq->opcap_bmap) {
@@ -1152,6 +1155,8 @@ static int idxd_wq_load_config(struct idxd_wq *wq) wq->max_xfer_bytes =1ULL << wq->wqcfg->max_xfer_shift;idxd_wq_set_max_batch_size (idxd->data->type, wq,1U << wq->wqcfg->max_batch_shift); +if (idxd_sgl_supported (idxd)) + wq->max_sgl_size =1U << wq->wqcfg->max_sgl_shift;for (i =0 ; i <WQCFG_STRIDES (idxd); i++) { wqcfg_offset =WQCFG_OFFSET (idxd, wq->id, i);
drivers/dma/idxd/idxd.h +16 -0
@@ -227,6 +227,7 @@ struct idxd_wq {char name[WQ_NAME_SIZE +1 ];u64 max_xfer_bytes;u32 max_batch_size; +u32 max_sgl_size;/* Lock to protect upasid_xa access. */ struct mutex uc_lock;
@@ -348,6 +349,7 @@ struct idxd_device {u64 max_xfer_bytes;u32 max_batch_size; +u32 max_sgl_size;int max_groups;int max_engines;int max_rdbufs;
@@ -692,6 +694,20 @@ static inline void idxd_wq_set_max_batch_size(int idxd_type, struct idxd_wq *wq, wq->max_batch_size = max_batch_size; } +static bool idxd_sgl_supported (struct idxd_device *idxd) +{ +return idxd->data->type ==IDXD_TYPE_DSA && + idxd->hw.version >=DEVICE_VERSION_3 && + idxd->hw.dsacap0.sgl_formats; +} + +static inline void idxd_wq_set_init_max_sgl_size (struct idxd_device *idxd, +struct idxd_wq *wq) +{ +if (idxd_sgl_supported (idxd)) + wq->max_sgl_size =1U << idxd->hw.dsacap0.max_sgl_shift; +} +static inline void idxd_wqcfg_set_max_batch_shift (int idxd_type,union wqcfg *wqcfg,u32 max_batch_shift) {
drivers/dma/idxd/init.c +5 -0
@@ -217,6 +217,7 @@ static int idxd_setup_wqs(struct idxd_device *idxd)init_completion (&wq->wq_resurrect); wq->max_xfer_bytes =WQ_DEFAULT_MAX_XFER ;idxd_wq_set_max_batch_size (idxd->data->type, wq,WQ_DEFAULT_MAX_BATCH ); +idxd_wq_set_init_max_sgl_size (idxd, wq); wq->enqcmds_retries =IDXD_ENQCMDS_RETRIES ; wq->wqcfg =kzalloc_node (idxd->wqcfg_size,GFP_KERNEL ,dev_to_node (dev));if (!wq->wqcfg) {
@@ -587,6 +588,10 @@ static void idxd_read_caps(struct idxd_device *idxd) idxd->hw.dsacap1.bits =ioread64 (idxd->reg_base +IDXD_DSACAP1_OFFSET ); idxd->hw.dsacap2.bits =ioread64 (idxd->reg_base +IDXD_DSACAP2_OFFSET ); } +if (idxd_sgl_supported (idxd)) { + idxd->max_sgl_size =1U << idxd->hw.dsacap0.max_sgl_shift; +dev_dbg (dev,"max sgl size: %u\n" , idxd->max_sgl_size); + }/* read iaa cap */ if (idxd->data->type ==IDXD_TYPE_IAX && idxd->hw.version >=DEVICE_VERSION_2 )
drivers/dma/idxd/registers.h +2 -1
@@ -385,7 +385,8 @@ union wqcfg {/* bytes 12-15 */ u32 max_xfer_shift:5; u32 max_batch_shift:4; - u32 rsvd4:23; + u32 max_sgl_shift:4; + u32 rsvd4:19;/* bytes 16-19 */ u16 occupancy_inth;
Signature · 1 line
2.43.0
Re: [PATCH v2 2/2] dmaengine: idxd: Add Max SGL Size Support for DSA3.0
Quoted 12 lines · Yi Sun
Certain DSA 3.0 opcodes, such as Gather copy and Gather reduce, require max SGL configured for workqueues prior to supporting these opcodes.
Configure the maximum scatter-gather list (SGL) size for workqueues during setup on the supported HW. Application can then properly handle the SGL size without explicitly setting it.
Signed-off-by Yi Sun <[email protected]>Co-developed-by Anil S Keshavamurthy <[email protected]>Signed-off-by Anil S Keshavamurthy <[email protected]>Reviewed-by Dave Jiang <[email protected]>
Cheers,
Signature · 1 line
Vinicius
Re: [PATCH v2 2/2] dmaengine: idxd: Add Max SGL Size Support for DSA3.0
Quoted 15 lines · Vinicius Costa Gomes
Quoted 12 lines · Yi Sun
Certain DSA 3.0 opcodes, such as Gather copy and Gather reduce, require max SGL configured for workqueues prior to supporting these opcodes.
Configure the maximum scatter-gather list (SGL) size for workqueues during setup on the supported HW. Application can then properly handle the SGL size without explicitly setting it.
Signed-off-by Yi Sun <[email protected]>Co-developed-by Anil S Keshavamurthy <[email protected]>Signed-off-by Anil S Keshavamurthy <[email protected]>Reviewed-by Dave Jiang <[email protected]>
Quoted 6 lines
Cheers,
Signature · 1 line
Vinicius
Hi Vinod,
Gentle ping on the entire series.
Thanks
--Sun, Yi
Vinod Koul
maintainerRe: [PATCH v2 2/2] dmaengine: idxd: Add Max SGL Size Support for DSA3.0
Quoted 19 lines · Yi Sun
Quoted 18 lines · Vinicius Costa Gomes
Quoted 12 lines · Yi Sun
Certain DSA 3.0 opcodes, such as Gather copy and Gather reduce, require max SGL configured for workqueues prior to supporting these opcodes.
Configure the maximum scatter-gather list (SGL) size for workqueues during setup on the supported HW. Application can then properly handle the SGL size without explicitly setting it.
Signed-off-by Yi Sun <[email protected]>Co-developed-by Anil S Keshavamurthy <[email protected]>Signed-off-by Anil S Keshavamurthy <[email protected]>Reviewed-by Dave Jiang <[email protected]>
Quoted 7 lines
Quoted 3 lines
Cheers,
Signature · 1 line
ViniciusHi Vinod,
Gentle ping on the entire series.
Can you please rebase and resend this
Signature · 1 line
~Vinod
RE: [PATCH v2 2/2] dmaengine: idxd: Add Max SGL Size Support for DSA3.0
Sure, you can expect it within a day. Some basic testing is still required.
Thanks
--Sun, Yi
-----Original Message-----
From: Vinod Koul <[email protected]>
Sent: Thursday, August 21, 2025 01:31
To: Sun, Yi <[email protected]>
Cc: Gomes, Vinicius <[email protected]>; Jiang, Dave <[email protected]>; [email protected]; [email protected]; [email protected]; Lantz, Philip <[email protected]>; Jin, Gordon <[email protected]>; [email protected]
Subject: Re: [PATCH v2 2/2] dmaengine: idxd: Add Max SGL Size Support for DSA3.0
Quoted 19 lines · Yi Sun
Quoted 18 lines · Vinicius Costa Gomes
Quoted 12 lines · Yi Sun
Certain DSA 3.0 opcodes, such as Gather copy and Gather reduce, require max SGL configured for workqueues prior to supporting these opcodes.
Configure the maximum scatter-gather list (SGL) size for workqueues during setup on the supported HW. Application can then properly handle the SGL size without explicitly setting it.
Signed-off-by Yi Sun <[email protected]>Co-developed-by Anil S Keshavamurthy <[email protected]>Signed-off-by Anil S Keshavamurthy <[email protected]>Reviewed-by Dave Jiang <[email protected]>
Quoted 7 lines
Quoted 3 lines
Cheers,
Signature · 1 line
ViniciusHi Vinod,
Gentle ping on the entire series.
Can you please rebase and resend this
Signature · 1 line
~Vinod