mlx5 PTM cross timestamping support
Subsystems (5)
Tariq Toukan
maintainerv3 cover mlx5 PTM cross timestamping support
Hi,
This is V3. You can find V2 as part of a larger series here:
https://lore.kernel.org/netdev/[email protected]/T/
This patchset by Rahul and Carolina adds PTM (Precision Time Measurement) support to the mlx5 driver.
PTM is a PCI extended capability introduced by PCI-SIG for providing an accurate read of the device clock offset without being impacted by asymmetric bus transfer rates.
The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.
Results:
1. No traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 125 ns | <29 ns | +-----+--------+--------+ | RTC | 248 ns | <34 ns | +-----+--------+--------+
2. With traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 254 ns | <40 ns | +-----+--------+--------+ | RTC | 255 ns | <45 ns | +-----+--------+--------+
Series generated against:
commit 1722389b0d86 ("Merge tag 'net-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net")
Thanks,
Tariq.
V3:
- Rebased on latest. As a result, had to replace the call to the recently removed function convert_art_ns_to_tsc().
- Added more CCs to the series per Jakub's feedback.
- Added perf numbers.
Carolina Jubran (1):
net/mlx5: Add support for enabling PTM PCI capability
Rahul Rameshbabu (2):
net/mlx5: Add support for MTPTM and MTCTR registers net/mlx5: Implement PTM cross timestamping support
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 1 + .../ethernet/mellanox/mlx5/core/lib/clock.c | 91 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ include/linux/mlx5/device.h | 7 +- include/linux/mlx5/driver.h | 2 + include/linux/mlx5/mlx5_ifc.h | 43 +++++++++ 6 files changed, 149 insertions(+), 1 deletion(-)
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v3 1/3 net/mlx5: Add support for MTPTM and MTCTR registers
From: Rahul Rameshbabu <[email protected]>
Make Management Precision Time Measurement (MTPTM) register and Management Cross Timestamp (MTCTR) register usable in mlx5 driver.
--- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 1 + include/linux/mlx5/device.h | 7 +++- include/linux/mlx5/driver.h | 2 + include/linux/mlx5/mlx5_ifc.h | 43 ++++++++++++++++++++ 4 files changed, 52 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +1 -0
@@ -224,6 +224,7 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)if (MLX5_CAP_GEN (dev, mcam_reg)) {mlx5_get_mcam_access_reg_group (dev,MLX5_MCAM_REGS_FIRST_128 );mlx5_get_mcam_access_reg_group (dev, MLX5_MCAM_REGS_0x9100_0x917F); +mlx5_get_mcam_access_reg_group (dev, MLX5_MCAM_REGS_0x9180_0x91FF); }if (MLX5_CAP_GEN (dev, qcam_reg))
include/linux/mlx5/device.h +6 -1
@@ -1243,7 +1243,8 @@ enum mlx5_pcam_feature_groups {enum mlx5_mcam_reg_groups {MLX5_MCAM_REGS_FIRST_128 =0x0 , MLX5_MCAM_REGS_0x9100_0x917F =0x2 , -MLX5_MCAM_REGS_NUM =0x3 , + MLX5_MCAM_REGS_0x9180_0x91FF =0x3 , +MLX5_MCAM_REGS_NUM =0x4 , };enum mlx5_mcam_feature_groups {
@@ -1392,6 +1393,10 @@ enum mlx5_qcam_feature_groups {MLX5_GET (mcam_reg, (mdev)->caps.mcam[MLX5_MCAM_REGS_0x9100_0x917F], \ mng_access_reg_cap_mask.access_regs2.reg) +#define MLX5_CAP_MCAM_REG3 (mdev, reg) \ + MLX5_GET(mcam_reg, (mdev)->caps.mcam[MLX5_MCAM_REGS_0x9180_0x91FF], \ + mng_access_reg_cap_mask.access_regs3.reg) +#define MLX5_CAP_MCAM_FEATURE (mdev, fld) \ MLX5_GET(mcam_reg, (mdev)->caps.mcam, mng_feature_cap_mask.enhanced_features.fld)
include/linux/mlx5/driver.h +2 -0
@@ -159,6 +159,8 @@ enum {MLX5_REG_MSECQ =0x9155 ,MLX5_REG_MSEES =0x9156 ,MLX5_REG_MIRC =0x9162 , +MLX5_REG_MTPTM =0x9180 , +MLX5_REG_MTCTR =0x9181 ,MLX5_REG_SBCAM =0xB01F ,MLX5_REG_RESOURCE_DUMP =0xC000 , MLX5_REG_DTOR = 0xC00E,
include/linux/mlx5/mlx5_ifc.h +43 -0
67 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH net-next V3 1/3] net/mlx5: Add support for MTPTM and MTCTR registers
Quoted 8 lines · Tariq Toukan
From: Rahul Rameshbabu <[email protected]>
Make Management Precision Time Measurement (MTPTM) register and Management Cross Timestamp (MTCTR) register usable in mlx5 driver.
Signed-off-by Rahul Rameshbabu <[email protected]>Signed-off-by Tariq Toukan <[email protected]>---
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 1 + include/linux/mlx5/device.h | 7 +++- include/linux/mlx5/driver.h | 2 + include/linux/mlx5/mlx5_ifc.h | 43 ++++++++++++++++++++ 4 files changed, 52 insertions(+), 1 deletion(-)
<...>
Tariq Toukan
maintainerPATCH v3 2/3 net/mlx5: Add support for enabling PTM PCI capability
In reply to Tariq ToukanFrom: Carolina Jubran <[email protected]>
Since the kernel doesn't support enabling Precision Time Measurement for an endpoint device, enable the PTM PCI capability in the driver.
--- drivers/net/ethernet/mellanox/mlx5/core/main.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/main.c +6 -0
@@ -923,6 +923,11 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct pci_dev *pdev, }mlx5_pci_vsc_init (dev); + + err =pci_enable_ptm (pdev,NULL ); +if (err) +mlx5_core_info (dev,"PTM is not supported by PCIe\n" ); +return 0 ; err_clr_master:
@@ -939,6 +944,7 @@ static void mlx5_pci_close(struct mlx5_core_dev *dev)* before removing the pci bars */ mlx5_drain_health_wq (dev); +pci_disable_ptm (dev->pdev);iounmap (dev->iseg);release_bar (dev->pdev);mlx5_pci_disable_device (dev);
Signature · 1 line
2.44.0
Re: [PATCH net-next V3 2/3] net/mlx5: Add support for enabling PTM PCI capability
Quoted 8 lines · Tariq Toukan
From: Carolina Jubran <[email protected]>
Since the kernel doesn't support enabling Precision Time Measurement for an endpoint device, enable the PTM PCI capability in the driver.
Signed-off-by Carolina Jubran <[email protected]>Signed-off-by Tariq Toukan <[email protected]>---
Quoted 27 lines
drivers/net/ethernet/mellanox/mlx5/core/main.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/main.c +6
-0@@ -923,6 +923,11 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct pci_dev *pdev, }mlx5_pci_vsc_init (dev); + + err =pci_enable_ptm (pdev,NULL ); +if (err) +mlx5_core_info (dev,"PTM is not supported by PCIe\n" ); +return 0 ; err_clr_master:@@ -939,6 +944,7 @@ static void mlx5_pci_close(struct mlx5_core_dev *dev)* before removing the pci bars */ mlx5_drain_health_wq (dev); +pci_disable_ptm (dev->pdev);iounmap (dev->iseg);release_bar (dev->pdev);mlx5_pci_disable_device (dev);
Tariq Toukan
maintainerPATCH v3 3/3 net/mlx5: Implement PTM cross timestamping support
In reply to Tariq ToukanFrom: Rahul Rameshbabu <[email protected]>
Expose Precision Time Measurement support through related PTP ioctl.
The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.
Results:
1. No traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 125 ns | <29 ns | +-----+--------+--------+ | RTC | 248 ns | <34 ns | +-----+--------+--------+
2. With traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 254 ns | <40 ns | +-----+--------+--------+ | RTC | 255 ns | <45 ns | +-----+--------+--------+
--- .../ethernet/mellanox/mlx5/core/lib/clock.c | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c +91 -0
@@ -38,6 +38,10 @@#include "lib/eq.h" #include "en.h" #include "clock.h" +#ifdef CONFIG_X86 +#include <linux/timekeeping.h> +#include <linux/cpufeature.h> +#endif /* CONFIG_X86 */ enum {MLX5_PIN_MODE_IN =0x0 ,
@@ -148,6 +152,87 @@ static int mlx5_set_mtutc(struct mlx5_core_dev *dev, u32 *mtutc, u32 size) MLX5_REG_MTUTC, 0, 1); } +#ifdef CONFIG_X86 +static bool mlx5_is_ptm_source_time_available (struct mlx5_core_dev *dev) +{ +u32 out[MLX5_ST_SZ_DW (mtptm_reg)] = {0 }; +u32 in[MLX5_ST_SZ_DW (mtptm_reg)] = {0 }; +int err; + +if (!MLX5_CAP_MCAM_REG3 (dev, mtptm)) +return false; + + err =mlx5_core_access_reg (dev, in,sizeof (in), out,sizeof (out),MLX5_REG_MTPTM , +0 ,0 ); +if (err) +return false; + +return !!MLX5_GET (mtptm_reg, out, psta); +} + +static int mlx5_mtctr_syncdevicetime (ktime_t *device_time, +struct system_counterval_t *sys_counterval, +void *ctx) +{ +u32 out[MLX5_ST_SZ_DW (mtctr_reg)] = {0 }; +u32 in[MLX5_ST_SZ_DW (mtctr_reg)] = {0 }; +struct mlx5_core_dev *mdev = ctx; +bool real_time_mode; +u64 host, device; +int err; + + real_time_mode =mlx5_real_time_mode (mdev); + +MLX5_SET (mtctr_reg, in, first_clock_timestamp_request, +MLX5_MTCTR_REQUEST_PTM_ROOT_CLOCK ); +MLX5_SET (mtctr_reg, in, second_clock_timestamp_request, + real_time_mode ?MLX5_MTCTR_REQUEST_REAL_TIME_CLOCK : +MLX5_MTCTR_REQUEST_FREE_RUNNING_COUNTER ); + + err =mlx5_core_access_reg (mdev, in,sizeof (in), out,sizeof (out),MLX5_REG_MTCTR , +0 ,0 ); +if (err) +return err; + +if (!MLX5_GET (mtctr_reg, out, first_clock_valid) || + !MLX5_GET (mtctr_reg, out, second_clock_valid)) +return -EINVAL ; + + host =MLX5_GET64 (mtctr_reg, out, first_clock_timestamp); + *sys_counterval = (struct system_counterval_t ) { + .cycles = host, + .cs_id =CSID_X86_ART , + .use_nsecs = true, + }; + + device =MLX5_GET64 (mtctr_reg, out, second_clock_timestamp); +if (real_time_mode) + *device_time =ns_to_ktime (REAL_TIME_TO_NS (device >>32 , device &U32_MAX )); +else + *device_time =mlx5_timecounter_cyc2time (&mdev->clock, device); + +return 0 ; +} + +static int mlx5_ptp_getcrosststamp (struct ptp_clock_info *ptp, +struct system_device_crosststamp *cts) +{ + struct mlx5_clock *clock = container_of(ptp, struct mlx5_clock, ptp_info); +struct system_time_snapshot history_begin = {0 }; +struct mlx5_core_dev *mdev; + + mdev = container_of(clock, struct mlx5_core_dev, clock); + +if (!mlx5_is_ptm_source_time_available (mdev)) +return -EBUSY ; + +ktime_get_snapshot (&history_begin); + +return get_device_system_crosststamp (mlx5_mtctr_syncdevicetime, mdev, + &history_begin, cts); +} +#endif /* CONFIG_X86 */ +static u64 mlx5_read_time (struct mlx5_core_dev *dev,struct ptp_system_timestamp *sts,bool real_time)
@@ -1034,6 +1119,12 @@ static void mlx5_init_timer_clock(struct mlx5_core_dev *mdev)if (MLX5_CAP_MCAM_REG (mdev, mtutc))mlx5_init_timer_max_freq_adjustment (mdev); +#ifdef CONFIG_X86 +if (MLX5_CAP_MCAM_REG3 (mdev, mtptm) && +MLX5_CAP_MCAM_REG3 (mdev, mtctr) &&boot_cpu_has (X86_FEATURE_ART )) + clock->ptp_info.getcrosststamp = mlx5_ptp_getcrosststamp; +#endif /* CONFIG_X86 */ +mlx5_timecounter_init (mdev);mlx5_init_clock_info (mdev);mlx5_init_overflow_period (clock);
Signature · 1 line
2.44.0
Re: [PATCH net-next V3 3/3] net/mlx5: Implement PTM cross timestamping support
Quoted 29 lines · Tariq Toukan
From: Rahul Rameshbabu <[email protected]>
Expose Precision Time Measurement support through related PTP ioctl.
The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.
Results:
1. No traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 125 ns | <29 ns | +-----+--------+--------+ | RTC | 248 ns | <34 ns | +-----+--------+--------+2. With traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 254 ns | <40 ns | +-----+--------+--------+ | RTC | 255 ns | <45 ns | +-----+--------+--------+
Quoted 5 lines
Signed-off-by Rahul Rameshbabu <[email protected]>Co-developed-by Carolina Jubran <[email protected]>Signed-off-by Carolina Jubran <[email protected]>Signed-off-by Tariq Toukan <[email protected]>---
<...>
Quoted 58 lines · Tariq Toukan
Hi,
This is V3. You can find V2 as part of a larger series here:
https://lore.kernel.org/netdev/[email protected]/T/This patchset by Rahul and Carolina adds PTM (Precision Time Measurement) support to the mlx5 driver.
PTM is a PCI extended capability introduced by PCI-SIG for providing an accurate read of the device clock offset without being impacted by asymmetric bus transfer rates.
The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.
Results:
1. No traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 125 ns | <29 ns | +-----+--------+--------+ | RTC | 248 ns | <34 ns | +-----+--------+--------+2. With traffic
+-----+--------+--------+ | | No-PTM | PTM | +-----+--------+--------+ | FRC | 254 ns | <40 ns | +-----+--------+--------+ | RTC | 255 ns | <45 ns | +-----+--------+--------+Series generated against:
commit1722389b0d86("Merge tag 'net-6.11-rc1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/netdev/net")Thanks,
Tariq.V3:
- Rebased on latest. As a result, had to replace the call to the recently removed functionconvert_art_ns_to_tsc().
- Added more CCs to the series per Jakub's feedback.
- Added perf numbers.Carolina Jubran (1):
net/mlx5: Add support for enabling PTM PCI capabilityRahul Rameshbabu (2):
net/mlx5: Add support for MTPTM and MTCTR registers net/mlx5: Implement PTM cross timestamping support
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 1 + .../ethernet/mellanox/mlx5/core/lib/clock.c | 91 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ include/linux/mlx5/device.h | 7 +- include/linux/mlx5/driver.h | 2 + include/linux/mlx5/mlx5_ifc.h | 43 +++++++++ 6 files changed, 149 insertions(+), 1 deletion(-)
For the series:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 9 lines · On Tue, 30 Jul 2024 16:40:51 +0300 you wrote:
Hi,
This is V3. You can find V2 as part of a larger series here:
https://lore.kernel.org/netdev/[email protected]/T/This patchset by Rahul and Carolina adds PTM (Precision Time Measurement) support to the mlx5 driver.
[...]
Here is the summary with links:
- [net-next,V3,1/3] net/mlx5: Add support for MTPTM and MTCTR registers
https://git.kernel.org/netdev/net-next/c/7e45c1e9edc0 - [net-next,V3,2/3] net/mlx5: Add support for enabling PTM PCI capability https://git.kernel.org/netdev/net-next/c/bec6d85d43eb - [net-next,V3,3/3] net/mlx5: Implement PTM cross timestamping support https://git.kernel.org/netdev/net-next/c/d17125fb0923
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html