ethtool: provide the dim profile fine-tuning channel
Subsystems (10)
v15 cover ethtool: provide the dim profile fine-tuning channel
The NetDIM library provides excellent acceleration for many modern network cards. However, the default profiles of DIM limits its maximum capabilities for different NICs, so providing a way which the NIC can be custom configured is necessary.
Currently, the way is based on the commonly used "ethtool -C".
For example,
on the server side, the virtio-net NIC with rx dim enabled has 8 queues and runs nginx.
The client uses the following command to send traffic to the server:
./wrk http://server_ip:80 -c 64 -t 5 -d 30
Then adjust the default rx-profile for server dim to
{.usec = 1, .pkts = 256, .comps = n/a,},
{.usec = 8, .pkts = 256, .comps = n/a,},
{.usec = 30, .pkts = 256, .comps = n/a,},
{.usec = 64, .pkts = 256, .comps = n/a,},
{.usec = 128, .pkts = 256, .comps = n/a,}
The server PPS is improved by 20%+.
Please review, thank you very much!
Changelog
=====v14->v15:
- Modify the mod bit and add some hints.
Jakub feedback: Use RESEND to refresh the review queue.
v13->v14:
- Make DIMLIB dependent on NET (patch 2/5).
v12->v13:
- Rebase net-next to fix the one-line conflict.
- Update tiny comments.
- Config ETHTOOL_NETLINK to select DIMLIB.
v11->v12:
- Remove the use of IS_ENABLED(DIMLIB).
- Update Simon's htmldoc hint.
v10->v11:
- Fix and clean up some issues from Kuba, thanks.
- Rebase net-next/main
v9->v10:
- Collect dim related flags/mode/work into one place.
- Use rx_profile + tx_profile instead of four profiles.
- Add several helps.
- Update commit logs.
v8->v9:
- Fix the compilation error of conflicting names of rx_profile in dim.h and ice driver: in dim.h, rx_profile is replaced with dim_rx_profile. So does tx_profile.
v7->v8:
- Use kmemdup() instead of kzalloc()/memcpy() in dev_dim_profile_init().
v6->v7:
- A new wrapper struct pointer is used in struct net_device.
- Add IS_ENABLED(CONFIG_DIMLIB) to avoid compiler warnings.
- Profile fields changed from u16 to u32.
v5->v6:
- Place the profile in netdevice to bypass the driver.
The interaction code of ethtool <-> kernel has not changed at all,
only the interaction part of kernel <-> driver has changed.v4->v5:
- Update some snippets from Kuba.
v3->v4:
- Some tiny updates and patch 1 only add a new comment.
v2->v3:
- Break up the attributes to avoid the use of raw c structs.
- Use per-device profile instead of global profile in the driver.
v1->v2:
- Use ethtool tool instead of net-sysfs.
Heng Qi (5):
linux/dim: move useful macros to .h file
dim: make DIMLIB dependent on NET
ethtool: provide customized dim profile management
dim: add new interfaces for initialization and getting results
virtio-net: support dim profile fine-tuning
Documentation/netlink/specs/ethtool.yaml | 31 +++ Documentation/networking/ethtool-netlink.rst | 8 + Documentation/networking/net_dim.rst | 42 +++ drivers/net/virtio_net.c | 54 +++- drivers/soc/fsl/Kconfig | 2 +- include/linux/dim.h | 113 ++++++++ include/linux/ethtool.h | 4 +- include/linux/netdevice.h | 3 + include/uapi/linux/ethtool_netlink.h | 22 ++ lib/Kconfig | 1 + lib/dim/net_dim.c | 144 +++++++++- net/Kconfig | 1 + net/ethtool/coalesce.c | 273 ++++++++++++++++++- 13 files changed, 681 insertions(+), 17 deletions(-)
Signature · 1 line
2.32.0.3.g01195cf9f
PATCH v15 1/5 linux/dim: move useful macros to .h file
Useful macros will be used effectively elsewhere.
These will be utilized in subsequent patches.
--- include/linux/dim.h | 7 +++++++ lib/dim/net_dim.c | 6 ------ 2 files changed, 7 insertions(+), 6 deletions(-)
include/linux/dim.h +7 -0
@@ -10,6 +10,13 @@#include <linux/types.h> #include <linux/workqueue.h> +/* Number of DIM profiles and period mode. */ +#define NET_DIM_PARAMS_NUM_PROFILES 5 +#define NET_DIM_DEFAULT_RX_CQ_PKTS_FROM_EQE 256 +#define NET_DIM_DEFAULT_TX_CQ_PKTS_FROM_EQE 128 +#define NET_DIM_DEF_PROFILE_CQE 1 +#define NET_DIM_DEF_PROFILE_EQE 1 +/* * Number of events between DIM iterations. * Causes a moderation of the algorithm run.
lib/dim/net_dim.c +0 -6
@@ -11,12 +11,6 @@* There are different set of profiles for RX/TX CQs. * Each profile size must be of NET_DIM_PARAMS_NUM_PROFILES */ -#define NET_DIM_PARAMS_NUM_PROFILES 5 -#define NET_DIM_DEFAULT_RX_CQ_PKTS_FROM_EQE 256 -#define NET_DIM_DEFAULT_TX_CQ_PKTS_FROM_EQE 128 -#define NET_DIM_DEF_PROFILE_CQE 1 -#define NET_DIM_DEF_PROFILE_EQE 1 -#define NET_DIM_RX_EQE_PROFILES { \ {.usec = 1, .pkts = NET_DIM_DEFAULT_RX_CQ_PKTS_FROM_EQE,}, \ {.usec = 8, .pkts = NET_DIM_DEFAULT_RX_CQ_PKTS_FROM_EQE,}, \
Signature · 1 line
2.32.0.3.g01195cf9f
PATCH v15 2/5 dim: make DIMLIB dependent on NET
In reply to Heng QiDIMLIB's capabilities are supplied by the dim, net_dim, and rdma_dim objects, and dim's interfaces solely act as a base for net_dim and rdma_dim and are not explicitly used anywhere else. rdma_dim is utilized by the infiniband driver, while net_dim is for network devices, excluding the soc/fsl driver.
In this patch, net_dim relies on some NET's interfaces, thus DIMLIB needs to explicitly depend on the NET Kconfig.
The soc/fsl driver uses the functions provided by net_dim, so it also needs to depend on NET.
--- drivers/soc/fsl/Kconfig | 2 +- lib/Kconfig | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
drivers/soc/fsl/Kconfig +1 -1
@@ -22,7 +22,7 @@ config FSL_GUTS config FSL_MC_DPIO tristate "QorIQ DPAA2 DPIO driver" - depends on FSL_MC_BUS + depends on FSL_MC_BUS && NET select SOC_BUS select FSL_GUTS select DIMLIB
lib/Kconfig +1 -0
@@ -623,6 +623,7 @@ config SIGNATURE config DIMLIB tristate + depends on NET help Dynamic Interrupt Moderation library. Implements an algorithm for dynamically changing CQ moderation values
Signature · 1 line
2.32.0.3.g01195cf9f
PATCH v15 3/5 ethtool: provide customized dim profile management
In reply to Heng QiThe NetDIM library, currently leveraged by an array of NICs, delivers excellent acceleration benefits. Nevertheless, NICs vary significantly in their dim profile list prerequisites.
Specifically, virtio-net backends may present diverse sw or hw device implementation, making a one-size-fits-all parameter list impractical. On Alibaba Cloud, the virtio DPU's performance under the default DIM profile falls short of expectations, partly due to a mismatch in parameter configuration.
I also noticed that ice/idpf/ena and other NICs have customized profilelist or placed some restrictions on dim capabilities.
Motivated by this, I tried adding new params for "ethtool -C" that provides a per-device control to modify and access a device's interrupt parameters.
Usage
========
The target NIC is named ethx.
Assume that ethx only declares support for rx profile setting (with DIM_PROFILE_RX flag set in profile_flags) and supports modification of usec and pkt fields.
1. Query the currently customized list of the device
$ ethtool -c ethx
...
rx-profile:
{.usec = 1, .pkts = 256, .comps = n/a,},
{.usec = 8, .pkts = 256, .comps = n/a,},
{.usec = 64, .pkts = 256, .comps = n/a,},
{.usec = 128, .pkts = 256, .comps = n/a,},
{.usec = 256, .pkts = 256, .comps = n/a,}
tx-profile: n/a2. Tune
$ ethtool -C ethx rx-profile 1,1,n_2,n,n_3,3,n_4,4,n_n,5,n
"n" means do not modify this field.
$ ethtool -c ethx
...
rx-profile:
{.usec = 1, .pkts = 1, .comps = n/a,},
{.usec = 2, .pkts = 256, .comps = n/a,},
{.usec = 3, .pkts = 3, .comps = n/a,},
{.usec = 4, .pkts = 4, .comps = n/a,},
{.usec = 256, .pkts = 5, .comps = n/a,}
tx-profile: n/a3. Hint
If the device does not support some type of customized dim profiles, the corresponding "n/a" will display.
If the "n/a" field is being modified, -EOPNOTSUPP will be reported.
--- Documentation/netlink/specs/ethtool.yaml | 31 +++ Documentation/networking/ethtool-netlink.rst | 8 + Documentation/networking/net_dim.rst | 42 +++ include/linux/dim.h | 58 ++++ include/linux/ethtool.h | 4 +- include/linux/netdevice.h | 3 + include/uapi/linux/ethtool_netlink.h | 22 ++ lib/dim/net_dim.c | 70 +++++ net/Kconfig | 1 + net/ethtool/coalesce.c | 273 ++++++++++++++++++- 10 files changed, 509 insertions(+), 3 deletions(-)
Documentation/netlink/specs/ethtool.yaml +31 -0
@@ -414,6 +414,26 @@ attribute-sets: name: combined-count type: u32 + - + name: irq-moderation + attributes: + - + name: usec + type: u32 + - + name: pkts + type: u32 + - + name: comps + type: u32 + - + name: profile + attributes: + - + name: irq-moderation + type: nest + multi-attr: true + nested-attributes: irq-moderation - name: coalesce attributes:
@@ -502,6 +522,15 @@ attribute-sets: - name: tx-aggr-time-usecs type: u32 + - + name: rx-profile + type: nest + nested-attributes: profile + - + name: tx-profile + type: nest + nested-attributes: profile + - name: pause-stat attributes:
@@ -1325,6 +1354,8 @@ operations: - tx-aggr-max-bytes - tx-aggr-max-frames - tx-aggr-time-usecs + - rx-profile + - tx-profile dump: *coalesce-get-op - name: coalesce-set
Documentation/networking/ethtool-netlink.rst +8 -0
@@ -1033,6 +1033,8 @@ Kernel response contents: ``ETHTOOL_A_COALESCE_TX_AGGR_MAX_BYTES`` u32 max aggr size, Tx ``ETHTOOL_A_COALESCE_TX_AGGR_MAX_FRAMES`` u32 max aggr packets, Tx ``ETHTOOL_A_COALESCE_TX_AGGR_TIME_USECS`` u32 time (us), aggr, Tx + ``ETHTOOL_A_COALESCE_RX_PROFILE`` nested profile of DIM, Rx + ``ETHTOOL_A_COALESCE_TX_PROFILE`` nested profile of DIM, Tx =========================================== ====== ======================= Attributes are only included in reply if their value is not zero or the
@@ -1062,6 +1064,10 @@ block should be sent. This feature is mainly of interest for specific USB devices which does not cope well with frequent small-sized URBs transmissions. +``ETHTOOL_A_COALESCE_RX_PROFILE`` and ``ETHTOOL_A_COALESCE_TX_PROFILE`` refer +to DIM parameters, see `Generic Network Dynamic Interrupt Moderation (Net DIM) +<https://www.kernel.org/doc/Documentation/networking/net_dim.rst>`_. + COALESCE_SET ============
@@ -1098,6 +1104,8 @@ Request contents: ``ETHTOOL_A_COALESCE_TX_AGGR_MAX_BYTES`` u32 max aggr size, Tx ``ETHTOOL_A_COALESCE_TX_AGGR_MAX_FRAMES`` u32 max aggr packets, Tx ``ETHTOOL_A_COALESCE_TX_AGGR_TIME_USECS`` u32 time (us), aggr, Tx + ``ETHTOOL_A_COALESCE_RX_PROFILE`` nested profile of DIM, Rx + ``ETHTOOL_A_COALESCE_TX_PROFILE`` nested profile of DIM, Tx =========================================== ====== ======================= Request is rejected if it attributes declared as unsupported by driver (i.e.
Documentation/networking/net_dim.rst +42 -0
48 lines not on this page · raw message
include/linux/dim.h +58 -0
82 lines not on this page · raw message
include/linux/ethtool.h +3 -1
@@ -284,7 +284,9 @@ bool ethtool_convert_link_mode_to_legacy_u32(u32 *legacy_u32,#define ETHTOOL_COALESCE_TX_AGGR_MAX_BYTES BIT(24)#define ETHTOOL_COALESCE_TX_AGGR_MAX_FRAMES BIT(25)#define ETHTOOL_COALESCE_TX_AGGR_TIME_USECS BIT(26) -#define ETHTOOL_COALESCE_ALL_PARAMS GENMASK(26, 0) +#define ETHTOOL_COALESCE_RX_PROFILE BIT(27) +#define ETHTOOL_COALESCE_TX_PROFILE BIT(28) +#define ETHTOOL_COALESCE_ALL_PARAMS GENMASK(28, 0)#define ETHTOOL_COALESCE_USECS \ (ETHTOOL_COALESCE_RX_USECS | ETHTOOL_COALESCE_TX_USECS)
include/linux/netdevice.h +3 -0
@@ -2401,6 +2401,9 @@ struct net_device {/** @page_pools: page pools created for this netdevice */ struct hlist_head page_pools;#endif + +/** @irq_moder: dim parameters used if IS_ENABLED(CONFIG_DIMLIB). */ +struct dim_irq_moder *irq_moder; };#define to_net_dev (d) container_of(d, struct net_device, dev)
include/uapi/linux/ethtool_netlink.h +22 -0
@@ -415,12 +415,34 @@ enum {ETHTOOL_A_COALESCE_TX_AGGR_MAX_BYTES ,/* u32 */ ETHTOOL_A_COALESCE_TX_AGGR_MAX_FRAMES ,/* u32 */ ETHTOOL_A_COALESCE_TX_AGGR_TIME_USECS ,/* u32 */ +/* nest - _A_PROFILE_IRQ_MODERATION */ +ETHTOOL_A_COALESCE_RX_PROFILE , +/* nest - _A_PROFILE_IRQ_MODERATION */ +ETHTOOL_A_COALESCE_TX_PROFILE ,/* add new constants above here */ __ETHTOOL_A_COALESCE_CNT,ETHTOOL_A_COALESCE_MAX = (__ETHTOOL_A_COALESCE_CNT -1 ) }; +enum { +ETHTOOL_A_PROFILE_UNSPEC , +/* nest, _A_IRQ_MODERATION_* */ +ETHTOOL_A_PROFILE_IRQ_MODERATION , + __ETHTOOL_A_PROFILE_CNT, +ETHTOOL_A_PROFILE_MAX = (__ETHTOOL_A_PROFILE_CNT -1 ) +}; + +enum { +ETHTOOL_A_IRQ_MODERATION_UNSPEC , +ETHTOOL_A_IRQ_MODERATION_USEC ,/* u32 */ +ETHTOOL_A_IRQ_MODERATION_PKTS ,/* u32 */ +ETHTOOL_A_IRQ_MODERATION_COMPS ,/* u32 */ + + __ETHTOOL_A_IRQ_MODERATION_CNT, +ETHTOOL_A_IRQ_MODERATION_MAX = (__ETHTOOL_A_IRQ_MODERATION_CNT -1 ) +}; +/* PAUSE */ enum {
lib/dim/net_dim.c +70 -0
82 lines not on this page · raw message
net/Kconfig +1 -0
@@ -508,6 +508,7 @@ config FAILOVER config ETHTOOL_NETLINK bool "Netlink interface for ethtool" + select DIMLIB default y help An alternative userspace interface for ethtool based on generic
net/ethtool/coalesce.c +271 -2
344 lines not on this page · raw message
Signature · 1 line
2.32.0.3.g01195cf9f
PATCH v15 4/5 dim: add new interfaces for initialization and getting results
In reply to Heng QiDIM-related mode and work have been collected in one same place, so new interfaces are added to provide convenience.
--- include/linux/dim.h | 48 ++++++++++++++++++++++++++++++++ lib/dim/net_dim.c | 68 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+)
include/linux/dim.h +48 -0
@@ -256,6 +256,54 @@ int net_dim_init_irq_moder(struct net_device *dev, u8 profile_flags,*/ void net_dim_free_irq_moder (struct net_device *dev); +/** +* net_dim_setting - initialize DIM's cq mode and schedule worker +* @dev: target network device +* @dim: DIM context +* @is_tx: true indicates the tx direction, false indicates the rx direction +*/ +void net_dim_setting (struct net_device *dev,struct dim *dim,bool is_tx); + +/** +* net_dim_work_cancel - synchronously cancel dim's worker +* @dim: DIM context +*/ +void net_dim_work_cancel (struct dim *dim); + +/** +* net_dim_get_rx_irq_moder - get DIM rx results based on profile_ix +* @dev: target network device +* @dim: DIM context +* +* Return: DIM irq moderation +*/ +struct dim_cq_moder +net_dim_get_rx_irq_moder (struct net_device *dev,struct dim *dim); + +/** +* net_dim_get_tx_irq_moder - get DIM tx results based on profile_ix +* @dev: target network device +* @dim: DIM context +* +* Return: DIM irq moderation +*/ +struct dim_cq_moder +net_dim_get_tx_irq_moder (struct net_device *dev,struct dim *dim); + +/** +* net_dim_set_rx_mode - set DIM rx cq mode +* @dev: target network device +* @rx_mode: target rx cq mode +*/ +void net_dim_set_rx_mode (struct net_device *dev,u8 rx_mode); + +/** +* net_dim_set_tx_mode - set DIM tx cq mode +* @dev: target network device +* @tx_mode: target tx cq mode +*/ +void net_dim_set_tx_mode (struct net_device *dev,u8 tx_mode); +/** * dim_on_top - check if current state is a good place to stop (top location) * @dim: DIM context
lib/dim/net_dim.c +68 -0
74 lines not on this page · raw message
Signature · 1 line
2.32.0.3.g01195cf9f
PATCH v15 5/5 virtio-net: support dim profile fine-tuning
In reply to Heng QiVirtio-net has different types of back-end device implementations. In order to effectively optimize the dim library's gains for different device implementations, let's use the new interface params to initialize and query dim results from a customized profile list.
--- drivers/net/virtio_net.c | 54 ++++++++++++++++++++++++++++++++++------ 1 file changed, 47 insertions(+), 7 deletions(-)
drivers/net/virtio_net.c +47 -7
@@ -2468,6 +2468,13 @@ static int virtnet_enable_queue_pair(struct virtnet_info *vi, int qp_index)return err; } +static void virtnet_cancel_dim (struct virtnet_info *vi,struct dim *dim) +{ +if (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_VQ_NOTF_COAL )) +return ; +net_dim_work_cancel (dim); +} +static int virtnet_open (struct net_device *dev) {struct virtnet_info *vi =netdev_priv (dev);
@@ -2494,7 +2501,7 @@ static int virtnet_open(struct net_device *dev)for (i--; i >=0 ; i--) {virtnet_disable_queue_pair (vi, i); -cancel_work_sync (&vi->rq[i].dim.work); +virtnet_cancel_dim (vi, &vi->rq[i].dim); }return err;
@@ -2670,7 +2677,7 @@ static int virtnet_rx_resize(struct virtnet_info *vi,if (running) {napi_disable (&rq->napi); -cancel_work_sync (&rq->dim.work); +virtnet_cancel_dim (vi, &rq->dim); } err =virtqueue_resize (rq->vq, ring_num, virtnet_rq_unmap_free_buf);
@@ -2931,7 +2938,7 @@ static int virtnet_close(struct net_device *dev)for (i =0 ; i < vi->max_queue_pairs; i++) {virtnet_disable_queue_pair (vi, i); -cancel_work_sync (&vi->rq[i].dim.work); +virtnet_cancel_dim (vi, &vi->rq[i].dim); }return 0 ;
@@ -4457,7 +4464,7 @@ static void virtnet_rx_dim_work(struct work_struct *work)if (!rq->dim_enabled) goto out; - update_moder =net_dim_get_rx_moderation (dim->mode, dim->profile_ix); + update_moder =net_dim_get_rx_irq_moder (dev, dim);if (update_moder.usec != rq->intr_coal.max_usecs || update_moder.pkts != rq->intr_coal.max_packets) { err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, qnum,
@@ -5157,6 +5164,36 @@ static void virtnet_tx_timeout(struct net_device *dev, unsigned int txqueue) jiffies_to_usecs(jiffies - READ_ONCE(txq->trans_start))); } +static int virtnet_init_irq_moder (struct virtnet_info *vi) +{ +u8 profile_flags =0 , coal_flags =0 ; +int ret, i; + + profile_flags |=DIM_PROFILE_RX ; + coal_flags |=DIM_COALESCE_USEC |DIM_COALESCE_PKTS ; + ret =net_dim_init_irq_moder (vi->dev, profile_flags, coal_flags, +DIM_CQ_PERIOD_MODE_START_FROM_EQE , +0 , virtnet_rx_dim_work,NULL ); + +if (ret) +return ret; + +for (i =0 ; i < vi->max_queue_pairs; i++) +net_dim_setting (vi->dev, &vi->rq[i].dim, false); + +return 0 ; +} + +static void virtnet_free_irq_moder (struct virtnet_info *vi) +{ +if (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_VQ_NOTF_COAL )) +return ; + +rtnl_lock (); +net_dim_free_irq_moder (vi->dev); +rtnl_unlock (); +} +static const struct net_device_ops virtnet_netdev = { .ndo_open = virtnet_open, .ndo_stop = virtnet_close,
@@ -5436,9 +5473,6 @@ static int virtnet_alloc_queues(struct virtnet_info *vi) virtnet_poll_tx, napi_tx ? napi_weight : 0); -INIT_WORK (&vi->rq[i].dim.work, virtnet_rx_dim_work); - vi->rq[i].dim.mode =DIM_CQ_PERIOD_MODE_START_FROM_EQE ; -sg_init_table (vi->rq[i].sg,ARRAY_SIZE (vi->rq[i].sg));ewma_pkt_len_init (&vi->rq[i].mrg_avg_pkt_len);sg_init_table (vi->sq[i].sg,ARRAY_SIZE (vi->sq[i].sg));
@@ -5867,6 +5901,10 @@ static int virtnet_probe(struct virtio_device *vdev)for (i =0 ; i < vi->max_queue_pairs; i++)if (vi->sq[i].napi.weight) vi->sq[i].intr_coal.max_packets =1 ; + + err =virtnet_init_irq_moder (vi); +if (err) + goto free; }#ifdef CONFIG_SYSFS
@@ -6018,6 +6056,8 @@ static void virtnet_remove(struct virtio_device *vdev)disable_rx_mode_work (vi);flush_work (&vi->rx_mode_work); +virtnet_free_irq_moder (vi); +unregister_netdev (vi->dev);net_failover_destroy (vi->failover);
Signature · 1 line
2.32.0.3.g01195cf9f
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Fri, 21 Jun 2024 18:13:48 +0800 you wrote:
The NetDIM library provides excellent acceleration for many modern network cards. However, the default profiles of DIM limits its maximum capabilities for different NICs, so providing a way which the NIC can be custom configured is necessary.
Currently, the way is based on the commonly used "ethtool -C".
[...]
Here is the summary with links:
- [net-next,v15,1/5] linux/dim: move useful macros to .h file
https://git.kernel.org/netdev/net-next/c/0e942053e4dc - [net-next,v15,2/5] dim: make DIMLIB dependent on NET https://git.kernel.org/netdev/net-next/c/b65e697a7c9e - [net-next,v15,3/5] ethtool: provide customized dim profile management https://git.kernel.org/netdev/net-next/c/f750dfe825b9 - [net-next,v15,4/5] dim: add new interfaces for initialization and getting results https://git.kernel.org/netdev/net-next/c/13ba28c5cd04 - [net-next,v15,5/5] virtio-net: support dim profile fine-tuning https://git.kernel.org/netdev/net-next/c/dcb67f6a9ead
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html