Skip to contentSkip to conversation

mlx5 PTM cross timestamping support

PATCH Applied Tariq Toukan 4 participants 9 messages Jul 30, 2024 – Aug 5, 2024

Subsystems (5)
lore
Share & tools
lore

Tariq Toukan

author maintainer
#0 lore raw

v3 cover mlx5 PTM cross timestamping support

Hi,

This is V3. You can find V2 as part of a larger series here:
https://lore.kernel.org/netdev/[email protected]/T/

This patchset by Rahul and Carolina adds PTM (Precision Time Measurement) support to the mlx5 driver.

PTM is a PCI extended capability introduced by PCI-SIG for providing an accurate read of the device clock offset without being impacted by asymmetric bus transfer rates.

The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.

Results:

1. No traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 125 ns | <29 ns |
+-----+--------+--------+
| RTC | 248 ns | <34 ns |
+-----+--------+--------+

2. With traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 254 ns | <40 ns |
+-----+--------+--------+
| RTC | 255 ns | <45 ns |
+-----+--------+--------+

Series generated against:
commit 1722389b0d86 ("Merge tag 'net-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net")

Thanks,
Tariq.

V3:
- Rebased on latest. As a result, had to replace the call to the recently removed function convert_art_ns_to_tsc().
- Added more CCs to the series per Jakub's feedback.
- Added perf numbers.

Carolina Jubran (1):
net/mlx5: Add support for enabling PTM PCI capability

Rahul Rameshbabu (2):
net/mlx5: Add support for MTPTM and MTCTR registers net/mlx5: Implement PTM cross timestamping support

 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |  1 +
 .../ethernet/mellanox/mlx5/core/lib/clock.c   | 91 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/main.c    |  6 ++
 include/linux/mlx5/device.h                   |  7 +-
 include/linux/mlx5/driver.h                   |  2 +
 include/linux/mlx5/mlx5_ifc.h                 | 43 +++++++++
 6 files changed, 149 insertions(+), 1 deletion(-)
Signature · 1 line
2.44.0
author maintainer
#1 lore raw

PATCH v3 1/3 net/mlx5: Add support for MTPTM and MTCTR registers

From: Rahul Rameshbabu <[email protected]>

Make Management Precision Time Measurement (MTPTM) register and Management Cross Timestamp (MTCTR) register usable in mlx5 driver.

---
 drivers/net/ethernet/mellanox/mlx5/core/fw.c |  1 +
 include/linux/mlx5/device.h                  |  7 +++-
 include/linux/mlx5/driver.h                  |  2 +
 include/linux/mlx5/mlx5_ifc.h                | 43 ++++++++++++++++++++
 4 files changed, 52 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +1 -0
index b61b7d966114..76ad46bf477d 100644
@@ -224,6 +224,7 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 	if (MLX5_CAP_GEN(dev, mcam_reg)) {
 		mlx5_get_mcam_access_reg_group(dev, MLX5_MCAM_REGS_FIRST_128);
 		mlx5_get_mcam_access_reg_group(dev, MLX5_MCAM_REGS_0x9100_0x917F);
+		mlx5_get_mcam_access_reg_group(dev, MLX5_MCAM_REGS_0x9180_0x91FF);
 	}
 
 	if (MLX5_CAP_GEN(dev, qcam_reg))
include/linux/mlx5/device.h +6 -1
index ba875a619b97..a94bc9e3af96 100644
@@ -1243,7 +1243,8 @@ enum mlx5_pcam_feature_groups {
 enum mlx5_mcam_reg_groups {
 	MLX5_MCAM_REGS_FIRST_128                    = 0x0,
 	MLX5_MCAM_REGS_0x9100_0x917F                = 0x2,
-	MLX5_MCAM_REGS_NUM                          = 0x3,
+	MLX5_MCAM_REGS_0x9180_0x91FF                = 0x3,
+	MLX5_MCAM_REGS_NUM                          = 0x4,
 };
 
 enum mlx5_mcam_feature_groups {
@@ -1392,6 +1393,10 @@ enum mlx5_qcam_feature_groups {
 	MLX5_GET(mcam_reg, (mdev)->caps.mcam[MLX5_MCAM_REGS_0x9100_0x917F], \
 		 mng_access_reg_cap_mask.access_regs2.reg)
 
+#define MLX5_CAP_MCAM_REG3(mdev, reg) \
+	MLX5_GET(mcam_reg, (mdev)->caps.mcam[MLX5_MCAM_REGS_0x9180_0x91FF], \
+		 mng_access_reg_cap_mask.access_regs3.reg)
+
 #define MLX5_CAP_MCAM_FEATURE(mdev, fld) \
 	MLX5_GET(mcam_reg, (mdev)->caps.mcam, mng_feature_cap_mask.enhanced_features.fld)
 
include/linux/mlx5/driver.h +2 -0
index a96438ded15f..9f42834f57c5 100644
@@ -159,6 +159,8 @@ enum {
 	MLX5_REG_MSECQ		 = 0x9155,
 	MLX5_REG_MSEES		 = 0x9156,
 	MLX5_REG_MIRC		 = 0x9162,
+	MLX5_REG_MTPTM		 = 0x9180,
+	MLX5_REG_MTCTR		 = 0x9181,
 	MLX5_REG_SBCAM		 = 0xB01F,
 	MLX5_REG_RESOURCE_DUMP   = 0xC000,
 	MLX5_REG_DTOR            = 0xC00E,
include/linux/mlx5/mlx5_ifc.h +43 -0

67 lines not on this page · raw message

Signature · 1 line
2.44.0
Reviewed-by on v3 1/3
#2 lore raw

Re: [PATCH net-next V3 1/3] net/mlx5: Add support for MTPTM and MTCTR registers

Quoted 8 lines · Tariq Toukan

From: Rahul Rameshbabu <[email protected]>

Make Management Precision Time Measurement (MTPTM) register and Management Cross Timestamp (MTCTR) register usable in mlx5 driver.

---

Quoted 6 lines
 drivers/net/ethernet/mellanox/mlx5/core/fw.c |  1 +
 include/linux/mlx5/device.h                  |  7 +++-
 include/linux/mlx5/driver.h                  |  2 +
 include/linux/mlx5/mlx5_ifc.h                | 43 ++++++++++++++++++++
 4 files changed, 52 insertions(+), 1 deletion(-)

<...>

author maintainer
#3 lore raw

PATCH v3 2/3 net/mlx5: Add support for enabling PTM PCI capability

In reply to Tariq Toukan

From: Carolina Jubran <[email protected]>

Since the kernel doesn't support enabling Precision Time Measurement for an endpoint device, enable the PTM PCI capability in the driver.

---
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/main.c +6 -0
index 527da58c7953..780078bd5b8c 100644
@@ -923,6 +923,11 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct pci_dev *pdev,
 	}
 
 	mlx5_pci_vsc_init(dev);
+
+	err = pci_enable_ptm(pdev, NULL);
+	if (err)
+		mlx5_core_info(dev, "PTM is not supported by PCIe\n");
+
 	return 0;
 
 err_clr_master:
@@ -939,6 +944,7 @@ static void mlx5_pci_close(struct mlx5_core_dev *dev)
 	 * before removing the pci bars
 	 */
 	mlx5_drain_health_wq(dev);
+	pci_disable_ptm(dev->pdev);
 	iounmap(dev->iseg);
 	release_bar(dev->pdev);
 	mlx5_pci_disable_device(dev);
Signature · 1 line
2.44.0
Reviewed-by on v3 2/3
#4 lore raw

Re: [PATCH net-next V3 2/3] net/mlx5: Add support for enabling PTM PCI capability

Quoted 8 lines · Tariq Toukan

From: Carolina Jubran <[email protected]>

Since the kernel doesn't support enabling Precision Time Measurement for an endpoint device, enable the PTM PCI capability in the driver.

---

Quoted 27 lines
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/main.c +6 -0
index 527da58c7953..780078bd5b8c 100644
@@ -923,6 +923,11 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct pci_dev *pdev,
 	}
 
 	mlx5_pci_vsc_init(dev);
+
+	err = pci_enable_ptm(pdev, NULL);
+	if (err)
+		mlx5_core_info(dev, "PTM is not supported by PCIe\n");
+
 	return 0;
 
 err_clr_master:
@@ -939,6 +944,7 @@ static void mlx5_pci_close(struct mlx5_core_dev *dev)
 	 * before removing the pci bars
 	 */
 	mlx5_drain_health_wq(dev);
+	pci_disable_ptm(dev->pdev);
 	iounmap(dev->iseg);
 	release_bar(dev->pdev);
 	mlx5_pci_disable_device(dev);
author maintainer
#5 lore raw

PATCH v3 3/3 net/mlx5: Implement PTM cross timestamping support

In reply to Tariq Toukan

From: Rahul Rameshbabu <[email protected]>

Expose Precision Time Measurement support through related PTP ioctl.

The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.

Results:

1. No traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 125 ns | <29 ns |
+-----+--------+--------+
| RTC | 248 ns | <34 ns |
+-----+--------+--------+

2. With traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 254 ns | <40 ns |
+-----+--------+--------+
| RTC | 255 ns | <45 ns |
+-----+--------+--------+
---
 .../ethernet/mellanox/mlx5/core/lib/clock.c   | 91 +++++++++++++++++++
 1 file changed, 91 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c +91 -0
index 0361741632a6..b306ae79bf97 100644
@@ -38,6 +38,10 @@
 #include "lib/eq.h"
 #include "en.h"
 #include "clock.h"
+#ifdef CONFIG_X86
+#include <linux/timekeeping.h>
+#include <linux/cpufeature.h>
+#endif /* CONFIG_X86 */
 
 enum {
 	MLX5_PIN_MODE_IN		= 0x0,
@@ -148,6 +152,87 @@ static int mlx5_set_mtutc(struct mlx5_core_dev *dev, u32 *mtutc, u32 size)
 				    MLX5_REG_MTUTC, 0, 1);
 }
 
+#ifdef CONFIG_X86
+static bool mlx5_is_ptm_source_time_available(struct mlx5_core_dev *dev)
+{
+	u32 out[MLX5_ST_SZ_DW(mtptm_reg)] = {0};
+	u32 in[MLX5_ST_SZ_DW(mtptm_reg)] = {0};
+	int err;
+
+	if (!MLX5_CAP_MCAM_REG3(dev, mtptm))
+		return false;
+
+	err = mlx5_core_access_reg(dev, in, sizeof(in), out, sizeof(out), MLX5_REG_MTPTM,
+				   0, 0);
+	if (err)
+		return false;
+
+	return !!MLX5_GET(mtptm_reg, out, psta);
+}
+
+static int mlx5_mtctr_syncdevicetime(ktime_t *device_time,
+				     struct system_counterval_t *sys_counterval,
+				     void *ctx)
+{
+	u32 out[MLX5_ST_SZ_DW(mtctr_reg)] = {0};
+	u32 in[MLX5_ST_SZ_DW(mtctr_reg)] = {0};
+	struct mlx5_core_dev *mdev = ctx;
+	bool real_time_mode;
+	u64 host, device;
+	int err;
+
+	real_time_mode = mlx5_real_time_mode(mdev);
+
+	MLX5_SET(mtctr_reg, in, first_clock_timestamp_request,
+		 MLX5_MTCTR_REQUEST_PTM_ROOT_CLOCK);
+	MLX5_SET(mtctr_reg, in, second_clock_timestamp_request,
+		 real_time_mode ? MLX5_MTCTR_REQUEST_REAL_TIME_CLOCK :
+		 MLX5_MTCTR_REQUEST_FREE_RUNNING_COUNTER);
+
+	err = mlx5_core_access_reg(mdev, in, sizeof(in), out, sizeof(out), MLX5_REG_MTCTR,
+				   0, 0);
+	if (err)
+		return err;
+
+	if (!MLX5_GET(mtctr_reg, out, first_clock_valid) ||
+	    !MLX5_GET(mtctr_reg, out, second_clock_valid))
+		return -EINVAL;
+
+	host = MLX5_GET64(mtctr_reg, out, first_clock_timestamp);
+	*sys_counterval = (struct system_counterval_t) {
+			.cycles = host,
+			.cs_id = CSID_X86_ART,
+			.use_nsecs = true,
+	};
+
+	device = MLX5_GET64(mtctr_reg, out, second_clock_timestamp);
+	if (real_time_mode)
+		*device_time = ns_to_ktime(REAL_TIME_TO_NS(device >> 32, device & U32_MAX));
+	else
+		*device_time = mlx5_timecounter_cyc2time(&mdev->clock, device);
+
+	return 0;
+}
+
+static int mlx5_ptp_getcrosststamp(struct ptp_clock_info *ptp,
+				   struct system_device_crosststamp *cts)
+{
+	struct mlx5_clock *clock = container_of(ptp, struct mlx5_clock, ptp_info);
+	struct system_time_snapshot history_begin = {0};
+	struct mlx5_core_dev *mdev;
+
+	mdev = container_of(clock, struct mlx5_core_dev, clock);
+
+	if (!mlx5_is_ptm_source_time_available(mdev))
+		return -EBUSY;
+
+	ktime_get_snapshot(&history_begin);
+
+	return get_device_system_crosststamp(mlx5_mtctr_syncdevicetime, mdev,
+					     &history_begin, cts);
+}
+#endif /* CONFIG_X86 */
+
 static u64 mlx5_read_time(struct mlx5_core_dev *dev,
 			  struct ptp_system_timestamp *sts,
 			  bool real_time)
@@ -1034,6 +1119,12 @@ static void mlx5_init_timer_clock(struct mlx5_core_dev *mdev)
 	if (MLX5_CAP_MCAM_REG(mdev, mtutc))
 		mlx5_init_timer_max_freq_adjustment(mdev);
 
+#ifdef CONFIG_X86
+	if (MLX5_CAP_MCAM_REG3(mdev, mtptm) &&
+	    MLX5_CAP_MCAM_REG3(mdev, mtctr) && boot_cpu_has(X86_FEATURE_ART))
+		clock->ptp_info.getcrosststamp = mlx5_ptp_getcrosststamp;
+#endif /* CONFIG_X86 */
+
 	mlx5_timecounter_init(mdev);
 	mlx5_init_clock_info(mdev);
 	mlx5_init_overflow_period(clock);
Signature · 1 line
2.44.0
Reviewed-by on v3 3/3
#6 lore raw

Re: [PATCH net-next V3 3/3] net/mlx5: Implement PTM cross timestamping support

Quoted 29 lines · Tariq Toukan

From: Rahul Rameshbabu <[email protected]>

Expose Precision Time Measurement support through related PTP ioctl.

The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.

Results:

1. No traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 125 ns | <29 ns |
+-----+--------+--------+
| RTC | 248 ns | <34 ns |
+-----+--------+--------+

2. With traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 254 ns | <40 ns |
+-----+--------+--------+
| RTC | 255 ns | <45 ns |
+-----+--------+--------+
Quoted 5 lines

---

<...>

Tested-by on the v3 series
#7 lore raw
In reply to Tariq Toukan
Quoted 58 lines · Tariq Toukan

Hi,

This is V3. You can find V2 as part of a larger series here:
https://lore.kernel.org/netdev/[email protected]/T/

This patchset by Rahul and Carolina adds PTM (Precision Time Measurement) support to the mlx5 driver.

PTM is a PCI extended capability introduced by PCI-SIG for providing an accurate read of the device clock offset without being impacted by asymmetric bus transfer rates.

The performance of PTM on ConnectX-7 was evaluated using both real-time (RTC) and free-running (FRC) clocks under traffic and no traffic conditions. Tests with phc2sys measured the maximum offset values at a 50Hz rate, with and without PTM.

Results:

1. No traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 125 ns | <29 ns |
+-----+--------+--------+
| RTC | 248 ns | <34 ns |
+-----+--------+--------+

2. With traffic

+-----+--------+--------+
|     | No-PTM | PTM    |
+-----+--------+--------+
| FRC | 254 ns | <40 ns |
+-----+--------+--------+
| RTC | 255 ns | <45 ns |
+-----+--------+--------+

Series generated against:
commit 1722389b0d86 ("Merge tag 'net-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net")

Thanks,
Tariq.

V3:
- Rebased on latest. As a result, had to replace the call to the recently removed function convert_art_ns_to_tsc().
- Added more CCs to the series per Jakub's feedback.
- Added perf numbers.

Carolina Jubran (1):
net/mlx5: Add support for enabling PTM PCI capability

Rahul Rameshbabu (2):
net/mlx5: Add support for MTPTM and MTCTR registers net/mlx5: Implement PTM cross timestamping support

Quoted 8 lines
  drivers/net/ethernet/mellanox/mlx5/core/fw.c  |  1 +
  .../ethernet/mellanox/mlx5/core/lib/clock.c   | 91 +++++++++++++++++++
  .../net/ethernet/mellanox/mlx5/core/main.c    |  6 ++
  include/linux/mlx5/device.h                   |  7 +-
  include/linux/mlx5/driver.h                   |  2 +
  include/linux/mlx5/mlx5_ifc.h                 | 43 +++++++++
  6 files changed, 149 insertions(+), 1 deletion(-)

For the series:

#8 lore raw
In reply to Tariq Toukan
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 9 lines · On Tue, 30 Jul 2024 16:40:51 +0300 you wrote:

Hi,

This is V3. You can find V2 as part of a larger series here:
https://lore.kernel.org/netdev/[email protected]/T/

This patchset by Rahul and Carolina adds PTM (Precision Time Measurement) support to the mlx5 driver.

[...]

Here is the summary with links:
- [net-next,V3,1/3] net/mlx5: Add support for MTPTM and MTCTR registers

    https://git.kernel.org/netdev/net-next/c/7e45c1e9edc0
  - [net-next,V3,2/3] net/mlx5: Add support for enabling PTM PCI capability
    https://git.kernel.org/netdev/net-next/c/bec6d85d43eb
  - [net-next,V3,3/3] net/mlx5: Implement PTM cross timestamping support
    https://git.kernel.org/netdev/net-next/c/d17125fb0923

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html